Devices for refractive field visualization

ABSTRACT

An apparatus according to an embodiment of the present invention enables measurement and visualization of a refractive field such as a fluid. An embodiment device obtains video captured by a video camera with an imaging plane. Representations of apparent motions in the video are correlated to determine actual motions of the refractive field. A textured background of the scene can be modeled as stationary, with a refractive field translating between background and video camera. This approach offers multiple advantages over conventional fluid flow visualization, including an ability to use ordinary video equipment outside a laboratory without particle injection. Even natural backgrounds can be used, and fluid motion can be distinguished from refraction changes. Embodiments can render refractive flow visualizations for augmented reality, wearable devices, and video microscopes.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/279,266, filed May 15, 2014, which claims the benefit of U.S.Provisional Application No. 61/980,521, filed on Apr. 16, 2014, and U.S.Provisional Application No. 61/823,580, filed on May 15, 2013. Theentire teachings of the above applications are incorporated herein byreference.

GOVERNMENT SUPPORT

This invention was made with government support under Contract No.N00014-10-1-0951 awarded by the U.S. Navy. The government has certainrights in the invention.

BACKGROUND OF THE INVENTION

Measurement and visualization of air and fluid movement is useful insuch areas as aeronautical engineering, combustion research, andballistics. Various techniques have been proposed for measuring andvisualizing fluid motion, and they can be based on computation ofoptical flow in a video using an assumption of brightness constancy.Previously proposed techniques have included sound tomography, DopplerLIDAR, and schlieren photography.

Two general techniques to visualize and measure fluid flow includeintroduction of a foreign substance and optical techniques. Inforeign-substance techniques, substances such as a dye, smoke, orparticles are introduced into a fluid and then tracked to deduce fluidmotion. Optical techniques can include schlieren photography, forexample, in which an optical setup is used to amplify deflections oflight rays due to refraction to make the fluid flow visible. Analternative to standard schlieren photography is background-orientedschlieren photography (or synthetic schlieren photography), in whichfluid flow between each frame of a video and an undistorted referenceframe is computed in an attempt to recover fluid motion.

SUMMARY OF THE INVENTION

Current techniques for visualizing and measuring fluid flow havemultiple disadvantages and limitations. For example, they are oftenbased on complicated or expensive setups or are limited to in-lab use.One example is the schlieren photography arrangement. Schlierenphotography relies on a source grid in the background of the subject ofthe photography, as well as a camera configuration that includes acutoff grade and lamps to illuminate the background source. Thus, it canbe difficult to apply this technique outside of the laboratory setup.Furthermore, even background-oriented schlieren photography, orsynthetic schlieren, relies on having a background in focus and a lightfield probe as large as the target fluid flow of interest. Moreover,foreign substance techniques require the cost and inconvenience ofseeding a fluid with particles, may be limited to special studies, andis not suitable for monitoring in everyday use.

Moreover, the core assumption of brightness constancy on which standardoptical flow techniques are based does not hold for a refractive fluid.Brightness constancy does not hold for refractive fluids becauseintensity changes occur due to both translation of the fluid andrefractive effects of the fluid motion. The failure of the brightnessconstancy assumption can lead to both incorrect magnitude of motion andincorrect direction of motion for standard fluid flow calculations.

Embodiments of the current invention can remedy these deficiencies byusing refractive flow calculations. Refraction flow calculations do notultimately depend upon an assumption of brightness constancy, butinstead use a principle of refractive constancy. Refractive constancycan be applied to extract measurements and visualizations of fluid flowusing standard video images, without laboratory setups, artificialbackgrounds, or seeding with foreign substances. Embodiments enablefluid flow visualization and measurement using either artificial ornatural backgrounds that are at least partially textured. Embodimentscan be used outside of a laboratory environment in such diverseapplications as airport and in-flight turbulence monitoring, monitoringfor leaks of fluids such as hydrocarbons, wind turbine optimization,civil engineering, augmented reality, virtual reality, aeronauticalengineering, combustion research, and ballistics.

In one embodiment, for example, video images are captured by a standardvideo camera, and standard optical flow techniques are applied, assumingthat frame-to-frame changes in intensity or phase are caused by motionsof the fluid alone. Afterward, the apparent motions derived from theoptical flow technique are used to estimate actual motions by applying arefractive constancy principle in which motions of a refractive fieldare considered to be an exclusive cause of the apparent motions. Atextured background is modeled as being a stationary background, and afluid is modeled as being a refractive field translating between thetextured background and the video camera. Two-dimensional velocities ofelements of the refractive field are calculated based upon the actualmotions, and the two-dimensional velocities are output for visualizationas an augmented reality scene in which a real-life scene is displayedand overlaid with velocity vectors showing the fluid motion. The displayis then used to monitor for turbulence, leaks, updrafts, or to optimizea wind farm based on fluid flow around wind turbines. The display maybe, for example, a standard video or computer screen display or anaugmented reality eyewear or virtual reality device.

A method and a corresponding apparatus according to an embodiment of theinvention include obtaining video captured by a video camera with animaging plane. The method can also include correlating, via a processor,over time, representations of motions in the video from frame to frameas a function of motion of a refractive field through which light from atextured background passes to the imaging plane. The textured backgroundcan be a natural background or an artificial background.

Correlating the representations of the apparent motions can includemodeling the textured background as a stationary background and therefractive field as translating between the textured background and thevideo camera. Correlating can further include considering the apparentmotions to be constant from one video frame to another video frame andestimating actual motions based on the determined representations of theapparent motions by assuming that motions of the refractive field are anexclusive cause of the apparent motions. The representations of themotions can be representations of apparent motions observed at theimaging plane, and the method can further include determining therepresentations of the apparent motions by assuming that frame-to-framechanges in at least one of intensity and phase are caused by the motionsalone.

The method can also include calculating a velocity of the refractivefield based upon the correlated representations of the motions. Themotion of the refractive field can be motion of a fluid, and changes inrefractive indices of the refractive field can arise from motion of oneor more refractive objects in the fluid, and the representations ofmotions can be representations of motions of the one or more refractiveobjects. The one or more refractive objects can include, for example, atleast one of a small particle, molecule, region of the fluid differingin temperature from a surrounding region, region of fluid differing inpressure from a surrounding region, droplet, bubble, exhaust,hydrocarbon, or volatile substance. The fluid may be a gas or liquid,for example.

The method can further include calculating, based on the correlating, afluid velocity vector field as a function of position in the refractivefield. The method can also include displaying a representation of thefluid velocity vector field and displaying the representation in anarrangement with a real-life scene to present an augmented realityscene. The method can include using the fluid velocity vector field tomonitor for a hydrocarbon leak or to monitor for at least one of a fluidturbulence, updraft, downdraft, and vortex.

An apparatus according to an embodiment of the invention includes memoryconfigured to store representations of motions in a video obtained froma video camera with an imaging plane. The apparatus can also include aprocessor configured to correlate, over time, the representations of themotions from frame to frame as a function of motion of a refractivefield through which light from a textured background passes to theimaging plane. The representations of the motions can be representationsof apparent motions observed at the imaging plane, and the processor canfurther be configured to correlate the representations of the apparentmotions through an assumption of frame-to-frame changes in at least oneof intensity and phase resulting from the motions alone. The processorcan be configured to correlate the representations of the apparentmotions by estimating actual motions based on an assumption thattranslations of the refractive field are an exclusive cause of theapparent motions. The processor can also be configured to correlateusing an approximation that the apparent motions are constant from onevideo frame to another video frame.

The apparatus can also include a display configured to present a screenview with a representation of the fluid velocity vector field therein.The display can be an augmented reality display configured to registerthe fluid velocity vector field with a real-life scene in which thefluid velocity vector field is located with respect to a viewingposition and angle of a user.

The processor can also be configured to check for a hydrocarbon leakbased on the fluid velocity vector field or to use the fluid velocityvector field to check for at least one of a fluid turbulence, updraft,downdraft, and vortex.

A non-transient computer readable medium according to an embodiment ofthe invention can be programmed with computer readable code that uponexecution by a processor causes the processor to obtain video capturedby a video camera with an imaging plane. Execution of the computerreadable code by the processor can also cause the processor tocorrelate, over time, representations of motions in the video from frameto frame as a function of motion of a refractive field through whichlight from a textured background passes to the imaging plane.

A method and corresponding apparatus according to an embodiment of theinvention can include obtaining videos captured by at least two videocameras having respective imaging planes. The method can also includecorrelating, by a processor, over space, representations of motions inthe at least two videos from frame to frame as a function of motion of arefractive field through which light passes from a textured backgroundto the respective imaging planes. The method can also includecalculating, based on the correlated representations of the motions, adepth of the refractive field through use of the at least two videos.

The method can also include determining, based upon the correlatedrepresentations, three-dimensional velocity of the refractive field. Thetextured background can be out of focus at at least one of the cameraimaging planes, and the imaging planes of the at least two video camerascan be parallel to each other.

Correlating the representations of motions in the at least two videoscan include matching representations of motions in one of the videos torepresentations of motions in another of the videos. Correlating therepresentations can also include assuming that for each given time andfor a given spatial position and the at least two videos, therepresentations of motion corresponding to the given time and spatialposition are the same from video to video. The textured background canbe a natural textured background.

The method can also include determining an uncertainty of at least oneof the depth of the refractive field and a velocity of the refractivefield calculated from the correlated representations. Determining theuncertainty can include weighting the representations of motions as afunction of variance of an optical flow related to the representationsof the motions. Determining the uncertainty can also include definingweighted representations of motions to be a logarithm of a covariancebetween a representation of motion in one video and a representation ofmotion in another video. Determining the uncertainty can also includeweighting the representations of motions as a function of the degree oftexturing of the textured background.

A method and corresponding apparatus according to an embodiment of theinvention can include modeling a representation of motion in a videocaptured by a video camera as arising from motion of a refractive fieldthrough which light passes from a stationary background to an imagingplane of the video camera. The method can also include determining anduncertainty in the representation of the motion by a processor. Themethod can further comprise displaying the representation of motion anddisplaying the uncertainties in the representation through at least oneof shading and color coding the representation of motion. Theuncertainties can be applied to single camera embodiments or multiplecamera embodiments.

Determining the uncertainty can include calculating a variance of therepresentation of motion and weighting the representation of motion as afunction of the variance. Determining the uncertainty can also includeweighting the representation of motion as a function of the degree oftexturing of the stationary background. Determining the uncertainty canalso include applying an L₂ to the norm to the representation of motionand calculating a covariance of a concatenation of the plurality ofrepresentations of motion. Determining the uncertainty can furtherinclude weighting the representation of motion as a function of alogarithm of a covariance of the representation of motion in the videocaptured by the video camera and an additional representation of themotion in an additional video captured by an additional video camera.

A system and corresponding method according to an embodiment of theinvention includes an imaging module configured to acquire video of afluid, the imaging module having an imaging surface. The system can alsoinclude a processor configured to correlate representations of motionsof the fluid in the video from frame to frame as a function of motion ofa refractive field through which light passes from a stationarybackground to the imaging surface. The system can also include a displaymodule configured to display motions of the fluid, wherein the displayedmotions are based upon the correlated representations of the motions.

The imaging module can include at least one of a camera, a video camera,and a microscope. The imaging module can be further configured to viewat least one of the following: an airfield, atmospheric turbulence, windfarm, biological process, chemical process, hydrocarbon pipeline,solution, and cell culture.

The display module can be a traditional display module (e.g., video orcomputer display module), virtual reality display module, or augmentedreality display module. The display module can be further configured todisplay the video of the fluid. The display module can be wearable. Thestationary background can be a textured stationary background.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1A is an image-based overview of refractive flow principles,including a comparison of results based on optical flow and those basedon refractive flow.

FIG. 1B is an overview of refractive flow similar to the overview inFIG. 1A, but with different target scenes and textured backgrounds.

FIG. 1C(a)-(f) illustrates use of refractive flow according toembodiment methods and apparatus to obtain velocity and depth of arefractive field.

FIG. 1D(a)-(d) illustrates results of using uncertainty measurements toimprove refractive flow measurements.

FIG. 2 illustrates use of an embodiment of the invention for anaugmented reality display for an airplane pilot.

FIG. 3A illustrates use of an embodiment device in air traffic control.

FIG. 3B illustrates use of an embodiment in a wearable device, namelyaugmented reality eyewear.

FIGS. 3C-3E illustrate various environments in which embodiments can beused for monitoring applications, including wind farms, civilengineering, and hydrocarbon leak monitoring.

FIG. 3F illustrates an embodiment apparatus that can be used to monitora chemical process.

FIG. 3G illustrates and embodiment apparatus including a videomicroscope to monitor a biological cell culture.

FIG. 4A is a block diagram illustrating an embodiment device.

FIG. 4B is a block diagram illustrating an embodiment apparatus thatincludes a display and an alarm signal.

FIG. 4C is a block diagram illustrating an embodiment apparatus fordetermining fluid flow in three dimensions.

FIGS. 5A and 5B illustrate example procedures for correlatingrepresentations of motions in a video and using the correlatedrepresentations.

FIG. 5C illustrates an example procedure for determining uncertainty incorrelated representations of motion.

FIG. 6A illustrates use of a brightness constancy principle in the priorart to track motion of a solid object.

FIG. 6B illustrates use of a refractive constancy principle according toembodiments of the invention to obtain fluid object flow or refractiveflow.

FIG. 6C illustrates use of a refractive constancy principle according toembodiments of the invention to obtain depth of a refractive field aswell as refractive flow.

FIG. 7 illustrates the results of refractive flow analysis using asingle camera to view various scenes.

FIGS. 8A-8E illustrate quantitative evaluation of a refractive flowmethod according to an embodiment of the invention.

FIGS. 9A-9D illustrate a controlled experiment performed to evaluatefluid flow velocities estimated using refractive flow method accordingto an embodiment of the invention.

FIG. 10 is illustrates results of fluid depth measurements based onstereo video sequences.

FIGS. 11A-11B illustrate the effect of background texturing on depthmeasurement results with and without uncertainty applied.

FIG. 12 illustrates quantities referred to in proving refractiveconstancy.

FIG. 13 illustrates quantities used in proving stereo refractiveconstancy.

DETAILED DESCRIPTION

A description of example embodiments of the invention follows.

The teachings of all patents, published applications and referencescited herein are incorporated by reference in their entirety.

Measuring and visualizing how air and fluid move is of great importance,and has been the subject of study in broad areas of science andtechnology, including aeronautical engineering, combustion research, andballistics. Special techniques may be necessary to visualize fluid flow,and an example is shown in FIG. 1A.

FIG. 1A shows images 102 of a burning candle 104. A plume 105 a of thecandle 104 is invisible to the naked eye and to standard videotechniques. The heat rising from the burning candle 104 causes small,invisible distortions of a textured background 106 seen behind thecandle 104. The small distortions of the background result from lightrays being refracted by the plume 105 a as they travel from thebackground 106 to a camera (not shown). As the air in the candle plume105 a moves, small changes in the refractive properties of the airappeared as small visual deformations, or motions, of the texturedbackground 106. These motions can be referred to as “wiggles” or“refraction wiggles.”

Multiple techniques have been proposed for visualizing or measuring suchsmall distortions in fluids, such as sound tomography, Doppler LIDAR andschlieren photography. Current techniques to visualize and measure fluidflow can be divided into two categories: those which introduce a foreignsubstance (dye, smoke, or particles) into the fluid; and those that usethe optical refractive index of the fluid.

Where foreign substances or markers are introduced, the motions of themarkers can be used to give a visualization of the fluid flow. Throughparticle image velocimetry (PIV), a quantitative measurement of the flowcan be recovered by tracking the particles introduced into the fluid.

In the optical technique category, schlieren photography is one method.Schlieren photography uses a calibrated, elaborate optical setup thatcan include a source grid placed behind a photography subject, a cutoffgrid incorporated into a special camera, and special lamps forillumination to amplify deflections of light rays due to refraction tomake the fluid flow visible. To attempt to remedy some of thelimitations of schlieren photography, the schlieren method has beensimplified in the past decade. In background-oriented schlieren (BOS)photography (also referred to as “synthetic schlieren”), thecomplicated, hard-to-calibrate optical setup of traditional schlierenimaging is replaced by a combination of a video of a fluid flow capturedin front of an in-focus textured background (such as the texturedbackground 106 shown in FIG. 1A) and optical flow calculations (furtherdescribed hereinafter). The refraction due to the fluid motion isrecovered by computing optical flow between each frame of the video andan undistorted reference frame. A resulting displacement field gives avisualization of the fluid flow.

Continuing to refer to FIG. 1A, this figure provides an example ofschlieren photography images, namely the center images 110. Thesynthetic schlieren photography images 110 were created by applyingoptical flow calculations to the raw input video frames 102. Theapplication of optical flow calculations is symbolized in FIG. 1A by thearrow 103 a. The resulting displacement field provides a visualizationof the fluid flow above the candle 104. The displacement field can bepresented visually in various ways, including the visible plume 105 b inthe schlieren photography images 110 and the apparent motion velocityvectors v(x,y) 114 (or representations of the motions or refractionwiggles) overlaid on the video frames 112. The velocity vectors v(x,y)114 are oriented in an xy plane with an x-axis 117 and y-axis 119.

However, previous techniques have multiple disadvantages. Introductionof foreign substances or markers with use of PIV requires additionalexpense and is not suitable for continuous monitoring or virtual realityapplications, for example. Even for optical techniques, setups can beexpensive and limited to laboratory use. A schlieren photography setup,as described above, is an example of such limitations.

BOS presents an additional issue, in that a textured background must bein focus. To address this issue, light-field BOS has been proposed. Inlight-field BOS, a textured background like the textured background 106can be replaced by a light field probe in certain instances. However,the light field probe comes at a cost of having to build a light fieldprobe as large as the fluid flow of interest. Besides, for manyapplications such as continuous monitoring and virtual reality, lightfield probes would be undesirable or even impossible to use in somecases.

Furthermore, simply computing optical flow in a video will not generallyyield the correct motions of refractive elements, because theassumptions involved in these computations do not hold for refractivefluids. Optical techniques such as schlieren photography rely on abrightness constancy assumption. Under the assumption of brightnessconstancy (explained further hereinafter), any changes in pixelintensities are assumed to be caused only by translation of thephotographed objects. However, even if the brightness constancy holdsfor solid, reflecting objects, brightness constancy does not typicallyhold for refractive fluids such as the plume 105 a-b of the candle 104in FIG. 1A. In the case of refractive fluids, changes in pixelintensities result from both translation and refraction of the fluid.Thus, optical flow does not yield actual motion of a refractive fluid,but rather how much an image projected through the refractive fluid ontothe imaging plane of a camera moves.

Disclosed herein is a novel method to measure and visualize fluid flow.Even complicated airflows, for example, can be measured, withoutelaborate optical equipment. Binocular videos are not necessarilyrequired, and two-dimensional (2D) fluid flow can be derived even fromstandard monocular videos. Embodiments disclosed herein apply arecognition that local changes in intensity may be locally invariant,and an observed scene can be modeled as being composed of a staticbackground occluded by refractive elements that bend light and movethrough space. Embodiments of the current invention include computingthe optical flow of the optical flow by re-deriving the optical flowequation for a proposed image formation model. As used herein, “video”denotes any format that includes a series of images of a scene overtime, and “video camera” denotes any instrument capable of capturingsuch a series of images. Video cameras can be camera configured tocapture a series of images at a standard 30 frames per second (fps), ora higher or lower speeds, or a camera that takes single photographs atsimilar rates, not necessarily at a constant frame rate.

Embodiments can rely on fluid flows of interest containing movingrefractive structures, which are almost always present in fluid flows ofinterest. Embodiment methods can track features that move with the fluidto recover the fluid flow (also called refractive flow, or calculated oractual fluid flow). Advantageously, fluid flow can be obtained withoutthe added cost of seeding a fluid with particles and are thusmarkerless. Embodiments can use the displacement field (the fieldresulting from an initial computation of optical flow) to constructfeatures to track and can be implemented outside a laboratory and evenon mobile devices.

Continuing with the description of FIG. 1A, the velocity vectors v(x,y)114 are referred to as apparent motion velocity vectors because, asdescribed above, simply applying optical flow based on brightnessconstancy fails due to detected intensities being caused by bothtranslation of the plume 105 and refraction imparted by the plume.Changes in the apparent motion velocity vectors between the frames 112a, b, c . . . can be used as an indicator of the motion of thetransparent fluid, and the disparity of these motion features betweenviewpoints of two cameras, for example, can be used to determine thedepth of the fluid. Optical flow may be applied again, as represented byan arrow 103 b, to obtain refractive flow according to embodiments ofthe invention. Optical flow is applied again by a processor, which isconfigured to correlate, over time, the representations of motions inthe video frames 112 from frame to frame as a function of motion of therefractive field of the plume 105.

Correlating representations of motion can be done in accordance withmethods described hereinafter in conjunction with FIGS. 5A and 5B, forexample. Correlating over time can include tracking or matchingobserved, apparent motions represented by, for example, the vectors 114,over the video frames 112 a, b, c, and so forth, for example.Representations of motions can additionally be correlated over space bytracking the representations over video frames from a single camera ormultiple cameras. The correlation can be done as a function of motion ofa refractive field where, in the correlation, the represented, observedare assumed to arise from motion of a refractive field, with abackground of the refractive field being stationary, for example.

Light from the textured background 106 passes through the candle plume105 to an imaging plane of a video camera (not shown). Optical flow isapplied to obtain apparent motion velocity vectors v(x,y) 114, and theapparent motion velocity vectors of the video frames 112 are correlatedthrough applying optical flow again to obtain refractive flow.

The results of refractive flow are shown in FIG. 1A in the form ofcalculated fluid flow velocity vectors u(x,y) 118 overlaid on videoframes 116. Together, the video frames 116 with the calculated fluidflow velocity vectors u(x,y) 118 constitute a representation of a scene,which varies over time in successive video frames 116 a, 116 b, 116 c,and so forth.

FIG. 1B is a refractive flow overview 120, similar to the refractiveflow overview 100 in FIG. 1A, showing results of applying examplemethods described hereinafter in fluid flow applications. Theapplications illustrated in FIG. 1B include wind visualization 125 a,landing 125 b of a plane 124, and takeoff 125 c of the plane 124. Eachapplication includes a different textured background 126. In the windvisualization 125 a, the textured background 126 a consists primarily ofa natural background including trees. In contrast, the texturedbackgrounds 126 b and 126 c contain many elements made by humans. Thus,FIG. 1B illustrates how a wide variety of different textured backgroundscan be used, including both natural and artificial elements, to obtainrefractive flow measurements with standard video images.

In FIG. 1B, the process of obtaining refractive flow is similar to thatillustrated in FIG. 1A. Namely, optical flow is applied to raw inputvideo frames 122 to obtain representations of apparent motion observedat the imaging plane of the camera (not shown). These representationsare presented in the form of video frames 132 with apparent motionvectors v(x,y) 134 overlaid the representations in the video frames 132are correlated by again applying optical flow to obtain refractive flowmeasurements, which are represented by the video frames 136 withcalculated fluid flow vectors u(x,y) 138 overlaid.

FIGS. 1C(a)-(f) illustrate application of refractive flow principles toobtain both fluid flow and depth of a refractive field. FIG. 1C(a) showsa video image obtained from a left-side video camera, and FIG. 1C(b)shows the same scene as in FIG. 1C(a), but from a right-side camera (notshown). Heat rising from two burning candles 104 causes smalldistortions of the background due to light rays refracting as theytravel from the background to the cameras, passing through the hot aircreated by the candles. FIGS. 1C(c) and (d) show visualizations of theheat disturbances obtained by the existing synthetic schlieren imagingfor the left- and right-side cameras, respectively. The syntheticschlieren images in FIGS. 1C(c) and (d) can be used to visualize thesmall disturbances and reveal the heat plumes, but schlieren imaging isunable to produce a measurement of the actual motion of the heat plumes.Under reasonable conditions described hereinafter, the refractionpatterns (or observed motions) move coherently with a refracting fluid,allowing one to measure accurately both two-dimensional motion of thefluid flow and depth of the fluid flow.

FIG. 1C(e) shows calculated fluid flow velocity vectors 118 that can beobtained from a monocular video. In this case, the vectors 118 wereobtained from images from the left-side camera. Further, as shown inFIG. 1C(f), the depth of the fluid flow can be obtained from a stereoimage sequence. FIG. 1C(f) shows a disparity between the position of theplume obtained from the left-side camera and the position obtained fromthe right-side camera, and the disparity can be used to determine thedepth of the refractive field.

FIGS. 1D(a)-(d) illustrate how refractive flow measurements can befurther improved by applying uncertainty principles described more fullyhereinafter. FIG. 1D(a) shows a raw image of the candles 104, and FIG.1D(b1) shows apparent motion velocity vectors obtained from applyingoptical flow. The apparent motion of a fluid that is visualized usingoptical flow can also be described as a “wiggle.” FIG. 1D(b2) shows aschlieren image produced according to existing methods. FIG. 1D(c)illustrates apparent motion velocity vectors depicting the heat plumesrising from the candles 104, and the apparent motion velocity vectorsare obtained using methods and devices according to embodiments of theinvention. FIG. 1D(d) shows the apparent motion velocity vectorsweighted, as described hereinafter more fully, to improve accuracyrefractive flow results even further.

FIG. 2 illustrates an application of embodiments of the invention thatincludes real-time visualization of turbulence that can be encounteredby an airplane 224. Disturbances such as the updraft 228 a and theturbulent airflow to 228 b create a refractive field 228 that could beencountered by the airplane 224. Light rays from a textured background226 pass through, and are refracted by, the updraft 228 a and turbulence228 b and are captured by a video camera 240 mounted on the airplane.Also within the field of view 242 of the video camera are variouselements of the textured background 226, including a landscape 226 a,horizon 226 b, cloud 226 c, and distant turbulent air 226 d. Otherelements of the textured background 226 can be useful at night,including city lights 226 e, moon 226 f, and stars 226 g. Thus, FIG. 2illustrates that a wide variety of different background elements cancontribute to a textured background.

Even where turbulent air 228 b in the refractive field 228 is beingmonitored and visualized in the foreground of the camera 240, forexample, distant background turbulent air 226 d is relatively constantin position compared with the foreground refractive field 228 and isuseful as part of the textured background 226. The video camera 240provides raw video images 244 to be stored in memory 246. A fluid flowprocessor 248 obtains the raw video images 244 and applies examplemethods described hereinafter to produce refractive flow measurements.The refractive flow measurements are output from the fluid flowprocessor 248 in the form of images 228 with calculated refractive flowvectors overlaid. The images 228 are presented in an display 250 to beseen by a pilot 252. The display 250 is updated continually to providethe pilot 252 with a real-time view of the refractive field 228, andthus the pilot 252 can see and attempt to avoid turbulence such as theupdraft 228 a and the turbulent air 228 b.

It should be noted that in other embodiments, the display 250 can be anaugmented reality display. For example, calculated fluid flow velocityvectors such as the vectors 118 in FIG. 1A can be projected onto thewindows of the airplane 224 so that the vectors are seen in conjunctionwith the visual view of the atmosphere in which the refractive field 228is located.

FIG. 3A illustrates another setting in which embodiments of theinvention can be used, namely in an air traffic control context. A videocamera 340 a is mounted on an air traffic control tower 354 and has avideo camera field of view 342 a. The field of view 342 a includes areasin which airplanes, such as an airplane 324, take off and land. In otherembodiments, the video camera 340 a is mounted near runways or in otherlocations where it is useful to monitor airflow.

As understood in aviation, airplanes can produce wake turbulence, suchas jetwash 328 a and wingtip vortex 328 b produced by the airplane 324.Images from the video camera 340 a can be used to provide a display 350a for an air traffic controller 352. Moreover, the video camera 340 cansee other air disturbances, such as wind 329. The embodiment of FIG. 3Athus illustrates how safety can be improved by using embodiments of theinvention to monitor in real-time for both conditions created byairplanes and also natural weather conditions that can affect aviation.A real-time display, such as the display 350 a, can be a much moreuseful to tool to identify these conditions than windsocks or the otherweather monitoring instruments currently used because the display 350 acan show a wide field of view and is not limited to point locations aswindsocks are.

FIG. 3B illustrates how embodiments of the invention can be used inconjunction with augmented reality eyewear 356. A video camera 340 b ismounted to the eyewear 356. The video camera 340 b has a field of view342 b. An augmented reality display image 350 b is generated in thewearer's eye by a projector (not shown). The display image 350 bincludes an image of the field of view 342 b with a representation offluid flow in the field overlaid on the image. The representation of thefluid flow can include calculated fluid flow velocity vectors, such asthe vectors 138 shown in FIG. 1B. However, in other embodiments, fluidflow is represented in an augmented reality display image by colorcoding or some other means, for example.

Augmented reality eyewear, such as the eyewear 356 shown in FIG. 3B, canbe useful in many applications in which it is useful to visualize fluidflow. Such situations can include hang gliding, for example, wherein areal-time knowledge of updrafts, downdrafts, and other air currents isimportant for safety and flight optimization. It should be noted thatalternative embodiments of the invention can include virtual realityeyewear in which a virtual reality display image is generated in thewearer's eye. In the case of virtual reality, the eyewear would beopaque, and a completely artificial image would be generated in thewearer's eye. This is in contrast to the augmented reality eyewear 356in FIG. 3B, in which the wearer can see outside the eyewear, and theview is simply augmented by an overlaid image 350 b. According toembodiments of the invention, either fixed video cameras such as thecamera 340 a in FIG. 3A or mobile cameras with eyewear like the eyewear356 in FIG. 3B can be used to visualize airflow or refractive flow in avariety of situations. Some of these situations are shown in FIGS.3C-3E.

FIG. 3C illustrates a wind farm 358 in which wind turbines 360 areplaced. Fixed or mobile video cameras can be used to obtain video imagesfrom the wind farm 358 to optimize output of wind turbines 360. Forexample, the wind turbines 360 can be positioned to maximize outputbased on refractive flow around each turbine.

FIG. 3D illustrates a city 362 in which there are various refractiveelements that can be visualized by embodiment apparatuses. Suchrefractive sources include a steam source 364 a, an exhaust source 364b, a heat and particulate matter source 364 c, and wind 364 d from awater body 366 near the city 362. A knowledge of such refractive sourcescan be used in a variety of ways in civil engineering and environmentalmonitoring applications, for example.

FIG. 4C is a block diagram illustrating a correlating apparatus 400 cthat can receive stereo representations 474 d of motions captured by twovideo cameras and process the representations to obtain a depth 430 ofthe refractive field 428. The light 427 from the textured background 426passes through the moving refractive field 428 and on to the imagingplane 470 of the video camera 440 and an additional imaging plane 470′of an additional video camera 440 private. The video camera 440 outputsrepresentations 440 c of motions in the captured video, and the videocamera 440′ outputs representations 474 c′ of the motions in the videofrom camera 440′. Both the representations 474 c and 474 c′ are includedin the stereo representations 474 d of the motions in the videos. Thestereo representations 470 d are stored in a correlating apparatus 400 cin a memory 446 c. The correlating apparatus 400 c also includes a 3-D(or depth) processor 448 c, which obtains the stereo representations 474d from the memory 446 c. The 3-D processor 448 c is configured tocorrelate, over space, the stereo representations 474 d in the twovideos from frame to frame as a function of the motion 429 a of therefractive field 428. Correlation of the stereo representations overspace can include matching an apparent motion feature observed I onelocation of the imaging plane 470 with the same apparent motion featureobserved in another location of the imaging plan 470′. The 3-D processor448 c also calculates, based on the correlated representations of themotions, a depth 430 of the refractive field. The 3-D processor 448 ccan optionally output depths for multiple points on the refractive field428.

The 3-D processor 448 c can optionally output calculated 2-D fluid flowvelocity vectors 438. An optional uncertainty processor 448 d calculatesuncertainties of fluid flow velocity measurements and depth measurementsaccording to embodiment methods and weights the measurements to outputweighted fluid flow velocity measurements and depth measurements 432.While the uncertainty processor 448 d is separate from the 3-D processor448 c in FIG. 4C, in other embodiments, a single processor performscorrelation, depth, and uncertainty calculations. The uncertaintyprocessor 448 d can be configured to determine an uncertainty in therepresentations of the motion or in the depth of the refractive fieldusing a variety of techniques described hereinafter. In particular,determining the uncertainty can include calculating a variance of therepresentations of motion and weighting the representations of motion asa function of the variance. Furthermore, the processor 448 d can weightthe representations of motion as a function of a degree of texturing ofthe textured background 426. In particular, in regions of greatertexturing, the measurements of motion, or representations of motion, canbe given greater weight. The processor 448 d can apply an L2 norm to therepresentation of motion or calculate a covariance of the concatenationof a plurality of representations of motion, as described hereinafter.The determination of uncertainty can include waiting the representations474 c and 474 c′ as a function of a logarithm of a covariance of therepresentations of motion, and the covariance can be covariance of therepresentations 474 c from video camera 440 and the additionalrepresentations 474 c′ from the video camera 440′. Furthermore, in otherembodiments that include display such as the display 450 in FIG. 4B, therepresentations of motion can be displayed with the uncertainties in therepresentations by any means available, including, for example, shadingand color coding the representations of motion.

FIG. 3E illustrates one example application for the petroleum industry.Namely, stationary or mobile cameras, such as those shown in FIGS.3A-3B, can be used to monitor a pipeline 366 for a hydrocarbon leak 368.The pipeline 366 can be in a refinery or in a drilling location or otherfield or distribution location, for example. Furthermore, any volatilesubstance, droplet, bubble, or other refractive substance or object canbe visualized where it differs in refractive properties from asurrounding fluid.

FIGS. 3F and 3G illustrate that embodiments of the invention are notlimited to outdoor or even to macroscopic refractive flow applications.FIG. 3F shows an embodiment of the invention used for monitoring achemical process 351. The chemical process 351 occurs in a chemicalprocessor structure 349 and is monitored by a video camera 340 bconfigured to acquire video images of the chemical process 351 occurringin a fluid. Images of the chemical process 351 are sent to a fluid flowprocessor 248, which correlates representations of fluid motions in thechemical process 351 to obtain refractive flow measurements. The fluidflow processor 248 provides measurements of fluid flow in the chemicalprocessor 349 and sends the measurements to a display 350 c for viewingby a worker 352.

Refractive flow in the chemical processor 349 can result from bubblesrising, fluid heating, chemical reactions, or any other process thatproduces fluid currents within the processor 349. The video camera 340 bis configured to capture light passing from an artificial texturedbackground 326 through a fluid in the chemical processor 349 to animaging plane (not shown) of the video camera 340 b. In a controlled,artificial environment such as the chemical processor 349 in FIG. 3F,the artificial textured background 326 is preferable because it isprovides superior texturing for visualizing refractive changes in thechemical processor 349 and is much easier to implement in a controlledsetting than in the outdoors. However, in other embodiments, a naturaltextured background, such as the textured background 126 a in FIG. 1B,can be used.

FIG. 3G illustrates an embodiment of the invention used to view fluidmotion in a biological cell culture 353. FIG. 3G illustrates thatembodiments of the invention can be used to visualize fluid flow even ona microscopic scale. The biological cell culture 353 is situated in thevideo microscope 343 and is viewed by a microscope objective lens 341,which is part of the video microscope 343. The microscope 343 alsoincludes a video camera 340 c that acquires images of the biologicalcell culture 353 viewed through the microscope objective lens 341 andcaptured at an imaging plane (not shown). A textured background (notshown) can include, for example, an artificial textured background suchas the background 326 in FIG. 3F. The textured background can be scaledappropriately to have features comparable in size to the fluid motion inthe biological cell culture 353. The video camera 340 c outputs imagesof the cell culture 353, or representations of motions observed in thecell culture 353, to the fluid flow processor 248. The processor 248correlates the representations as a function of motion of the refractivefield in the cell culture 353. The correlated representations of themotions in the fluid are provided to a display 350 c for viewing ascalculated fluid flow vectors or in any other convenient form.

Lighting (not shown) in the video microscope 343 can be adjusted tooptimize illumination of the artificial textured background. Asunderstood in the art of microscopy, either forward lighting orbacklighting can be used, as deemed appropriate. It should also be notedthat, although not shown in FIGS. 3F and 3G, dual video cameras could beused in conjunction with the devices shown to obtain depth measurementsof the refractive field in the chemical processor 349 or in thebiological cell culture 353, as previously described in conjunction withFIGS. 1C(a)-(f), and as more fully described hereinafter.

FIG. 4A illustrates a correlating apparatus 400 a including memory 446 aand a correlation processor 448 a. The memory 446 a is configured tostore representations 474 a of motions in a video. The video is obtainedfrom a video camera 440 with an imaging plane 470. The correlatingapparatus 400 a also includes a processor 448 a that is configured tocorrelate, over time, the representations 474 a of the motions in thevideo that are stored in the memory 446 a. The processor 448 a isconfigured to correlate the representations from frame to frame as afunction of motion 429 a of a refractive field 428 through which light427 from a textured background 426 passes to the imaging plane 470 ofthe video camera 440. One way to perform this correlation is to applythe two-step refractive flow method described hereinafter. Thecorrelation processor 448 a is configured to output correlatedrepresentations 418. The correlated representations 418 are an optionalexample output, and the correlation processor 448 a outputs other typesof outputs in other embodiments. The correlated representations 418 canbe output in the form of calculated fluid flow velocity vectors, such asthe velocity vectors 118 in FIG. 1A, for example.

In FIG. 4A, the representations 474 a are in the form of electronicallystored apparent motion velocity vectors v(x,y) stored for various timesfor different video frames, such as the vectors 114 in FIG. 1A. However,in the other embodiments, the representations stored in the memory canbe in any form that permits the processor to correlate, over time, therepresentations from frame to frame as a function of motion of therefractive field. These representations can include raw video images,for example. Either of those forms, and other forms of representationsof the motions not shown here, can be suitable for correlation of therepresentations of the motions by the processor 448 a with appropriateadjustments to correlation methods. In particular, a variety ofelectronic formats can be used to store the images showing the motionsas understood in digital photography. Furthermore, as described earlierin conjunction with FIG. 1A, and as more fully described hereinafter,one convenient form of representations of motions in the video includesapparent motion velocity vectors such as the vectors 114 shown in FIG.1A and obtained by initially applying optical flow. Moreover, while thememory 446 a is shown as part of the correlating apparatus 400 a in FIG.4A, in other embodiments, the memory 446 a is incorporated into thevideo camera 440.

The correlation processor 448 a is configured to output correlatedrepresentations 418 of motions in the video. The correlatedrepresentations 418 are an example output from the processor and can bein the form of calculated fluid flow velocity vectors such as thevectors 118 in FIG. 1A, for example.

FIG. 4B is a block diagram illustrating an alternative correlatingapparatus 400 b. The apparatus 400 b includes several optional featuresnot included in the apparatus 400 a in FIG. 4A. The memory 446 b andprocessor for 448 b in correlating apparatus 400 b also are configuredto operate somewhat differently from those in the apparatus 400 a ofFIG. 4A. The memory 446 b in FIG. 4B for 446 b and processor 448 b arepart of a computer 472. The memory 446 b is configured to receive rawvideo images 444 from the video camera 440. Raw video frames 402 aretransferred from the memory 446 b to the processor 448 b for performinginitial optical flow calculations. The processor 448 b then transfersrepresentations 474 b of motions in the raw video to the memory 446 b inthe form of apparent motion velocity vectors, such as those representedgraphically in FIG. 1A as vectors 114. The processor 448 b then obtainsthe apparent motion velocity vectors 114 from the memory 446 b to applyoptical flow calculations again to obtain refractive flow measurements.Refractive flow measurements are then output by the processor 448 b to adisplay 450 in the form of calculated fluid flow velocity vectors 438overlaid on an image of the scene to present an augmented reality scene416.

The output from the processor 448 b to the display 450 can be via adedicated graphics card, for example. The processor 448 b also outputsan alarm signal 476 to an alarm 478. The alarm signal 476 can be outputto the alarm 478 via a driver (not shown), for example. The alarm 478 isactivated by the alarm signal 476 when at least one calculated fluidflow velocity vector exceeds a given threshold in magnitude. Thus, asthe alarm 478 feature demonstrates, displays such as the display 450having augmented reality scenes are not the only way to use refractiveflow measurements. The refractive flow measurements can be, optionally,further processed, displayed, or used in any way desired to visualize orcalculate or represent fluid flow.

FIG. 4B also illustrates that the refractive field 428 generallyincludes many motions 429 b, 429 c, and 429 d, for example, throughoutthe field. The motions 429 b-d of the refractive field can arise frommotion of one or more refractive objects in a fluid. Furthermore, therepresentations 474 b of motions in the raw video can be representationsof motions of the one or more refractive objects. Refractive objects caninclude smoke particles, molecules, regions of fluid differing intemperature from a surrounding region, regions of a fluid differing inpressure from surrounding region, droplets such as rain or condensation,bubbles such as bubbles in a liquid, exhaust, hydrocarbons, volatilesubstances, or any other refractive elements or substances in a fluid.

The 3-D processor 448 c is configured to correlate, over space, therepresentations 474 c and 474 c′ of motions from the two respectivevideo cameras 440 and 440′ with the respective imaging planes 470 and470′. The 3-D processor 448 c correlates the representations over spacefrom frame to frame as a function of the motion 429 a of the refractivefield 428. The representations of motions 474 c and 474 c′ arecorrelated over space when the location of a representation 474 cobserved at the imaging plane 470 is matched to a location on theimaging planes 470′ where a corresponding representation 474 c′ isobserved. Based on the correlated representations of the motions, adepth of the refractive field 428, or a point thereon, can becalculated, as described more fully hereinafter.

FIG. 5A illustrates a procedure 500 a for obtaining fluid flow orrefractive flow measurements. At 580 a, video captured by a video camerawith an imaging plane is obtained. At 580 b, representations of motionsin the video are correlated over time by a processor. Therepresentations are correlated from frame to frame as a function ofmotion of a refractive field through which light from a texturedbackground passes to the imaging plane of the video camera.

FIG. 5B illustrates an alternative procedure 500 b for obtaining fluidflow measurements. Referring to FIG. 5B, at 582 a, video captured by avideo camera with an imaging plane is obtained. At 582 b,representations of motions in the video obtained are defined to berepresentations of apparent motions observed at the imaging plane. Therepresentations of the apparent motions are also determined by assumingthat frame-to-frame changes in at least one of the intensity and phaseare caused by the motions alone. At 582 c, the representations ofmotions in the video are correlated over time from frame to frame as afunction of motion of a refractive field through which light from atextured background passes.

In FIG. 5B, the correlation at 582 c includes sub-elements 582 c 1, 582c 2, and 582 c 3. At 582 c 1, the textured background is modeled as astationary background, and the refractive field is modeled astranslating between the textured background and the video camera. At 582c, the apparent motions are considered to be constant from one videoframe to another video frame. At 582 c 3, actual motions of therefractive field are estimated based on the determined representationsof the apparent motions by assuming that motions of the refractive fieldare an exclusive cause of the apparent motions.

Referring to FIG. 5B, at 582 d, a velocity of the refractive field iscalculated based upon the correlated representations of the motions. At582 e, a fluid velocity vector field is calculated, based on thecorrelating, as a function of position in the refractive field.

At 582 g, a representation of the fluid velocity vector field isdisplayed. The display can be like the display 450 in FIG. 4B, forexample. At 582 h, the representation of the fluid velocity vector fieldis displayed in an arrangement with the real-life scene to present anaugmented reality scene. The augmented reality scene can be in the formof video frames like the video frames 116 with the calculated fluid flowvelocity vectors 118, as shown in FIG. 1A, for example. Alternatively,the augmented reality scene can be in the form of images such as theimages 228 shown in FIG. 2 on the display 250 or such as the augmentedreality display image 350 b generated in the wearer's eye in FIG. 3B.Furthermore, in other embodiments, a virtual reality scene can begenerated instead of an augmented reality scene.

Continuing to refer to FIG. 5B, at 584, the fluid velocity vector fieldis used to monitor for a hydrocarbon leak, a fluid turbulence, updraft,downdraft, or vortex. As shown in FIG. 5B, 584 can occur aftercalculating the fluid velocity vector field at 582 e. However, as alsoshown by a dashed line, in other embodiments, the fluid velocity vectorfield is used for monitoring at 584 after displaying an augmentedreality scene at 582 h. This difference in when the fluid velocityvector field is used for monitoring relates to whether or not anaugmented reality scene is used for display purposes. As explained inconjunction with FIG. 4B, the fluid velocity vector field may be used inan augmented reality display such as the display 450, or the fluidvelocity vector field may be used only for further calculations ormonitoring performed by a computer, for example.

One example of monitoring performed by a computer is found in FIG. 4B,wherein the processor 448 b analyzes the fluid velocity vector field todetermine whether any velocity exceeds a given threshold. If thethreshold is exceeded, the processor sends an alarm signal. In this way,use of the fluid velocity vector field can be by means of a computerand/or further calculations and need not be used to present an augmentedreality or virtual reality scene for viewing by a person.

FIG. 5C illustrates a procedure 500 c for determining an uncertainty inrefractive flow measurements. At 581 a, a representation of motion in avideo captured by video camera is modeled as arising from motion of arefractive field through which light passes from a stationary backgroundto an imaging plane of the video camera. At 581 b, uncertainty in therepresentation of the motion is determined by a processor.

Determining the uncertainty in the representation can includecalculating variance of the representation of motion and weighting therepresentation of motion as a function of the variance. In someembodiments, determining the uncertainty includes weighting therepresentation of motion as a function of the degree of texturing of thestationary background. For example, where the background is heavilytextured, the wiggles or representations of motion observed by a videocamera can be determined with greater certainty. On the other hand,where a background is only lightly textured, the representations ofmotion can be determined with less certainty, and these representationscan be downweighted in determining refractive flow. Furthermore, wherethe representations of motion or refractive flow measurements determinedthere from are displayed on a monitor or other image, uncertainty in therepresentations can also be displayed through, for example, shading orcolor coding the representations of motion. As described more fullyhereinafter, determining the uncertainty can include applying an L2 normto the representation of motion, calculating a covariance of theconcatenation of a plurality of representations of motion, or weightingthe representation of motion as a function of the logarithm of acovariance of the representation of motion in the video captured by thevideo camera and an additional representation of the motion in anadditional video captured by an additional video camera, such as thevideo cameras 440 and 440′ in FIG. 4C.

Refractive Fluid Flow

An example goal of refractive flow is to recover the projected 2Dvelocity, u(x, y, t), of a refractive fluid such as hot air or gas, froman observed image sequence, I(x, y, t). Before proceeding by analogy toa differential analysis of refraction flow, a differential analysis ofstandard optical flow for solid objects is first presented hereinafter.

Differential Analysis of Optical Flow

FIG. 6A includes a diagram 600 a illustrating application of abrightness constancy assumption to flow of a solid object 627 accordingto known methods. As seen in FIG. 6A, the solid object 627 moves withtime from one position at time t₁ to another position at time t₂. Anobject point 629 a on the solid object 627 also moves between times t₁and t₂. The object point 629 a is imaged to a projection point 629 a′ onthe camera imaging plane 670 at times t₁ and t₂, with a center ofprojection 690 for the video camera. An apparent motion velocity vector614 a represents the motion of the object point 629 a between times t₁and t₂ as observed at the camera imaging plane 670. Because light raysdo not penetrate the solid object 627, the textured background 426,shown for purposes of comparison with other figures, is not relevant inFIG. 6A.

Under the “brightness constancy” assumption, any changes in pixelintensities, I, are assumed to be caused by a translation with velocityv=(v_(x), v_(y)) over spatial horizontal or vertical positions, x and y,of the image intensities, where v_(x) and v_(y) are the x and ycomponents of the velocity, respectively. That is,

I(x, y, t)=I(x+Δx, y+Δy, t+t),  (1)

where (Δx, Δy) denotes the displacement of the point (x, y) at timet+Δt.

Applying a first order Taylor approximation to Equation 1 gives

$\begin{matrix}{{{\frac{\partial I}{\partial x}v_{x}} + {\frac{\partial I}{\partial y}v_{y}} + \frac{\partial I}{\partial t}} = 0.} & (2)\end{matrix}$

The velocity v can then be solved for by using optical flow methods suchas the Lucas-Kanade or Horn-schunck methods known in the art.

Extension to Refraction Flow

Brightness constancy, however, does not hold for refractive fluids, asrays passing through a point on the fluid intersect the background atdifferent points at different times, as shown in FIG. 6B.

FIG. 6B includes a diagram 600 b illustrating application of arefractive constancy principle to flow of a refractive fluid 628according to embodiments of the invention. In contrast to a solid object627, the refractive fluid 628 can transmit at least some light from thetextured background 426 through the refractive fluid 628 to the cameraimaging plane 670. Thus, as described previously, motion observed at thecamera imaging plane 670 arises both from motion of the refractive fluid628 and from refraction of light at the refractive fluid 628 during itsmotion. The textured background 426 can be a natural background, forexample, in that the textured background 426 can include natural oreveryday objects and need not be a special laboratory setup. Forexample, the textured background 426 can be like the texturedbackgrounds 126A-C in FIG. 1B or light the textured background 226 inFIG. 2, for example.

The refractive fluid 628 is shown at two time instants t₁ and t₂. Attime t₁, the fluid 628 refracts light rays, effectively movingbackground points 631 and 633 on the background to different positions,or projection points 629 b′, on the camera imaging plane 670. At timet₁, for example, the background point 631 appears at a differentposition on the imaging plane 670 than it does it at the time t₁+Δt.Similarly, at time t₂, the background point 633 appears at a differentpoint on the camera imaging plane 670 than it does at time t₂+Δt.

In FIG. 6B, the observed motions on the imaging plane are referred to asapparent motion velocity vectors 614 b 1 and 614 b 2. In general, thedirection of the observed motions on the imaging plane due to therefraction can be arbitrary. In the case illustrated in FIG. 6B, theobserved motions or apparent motions 614 b 1 and 614 b 2 are in the samedirection as the actual or calculated fluid flow, as shown by acalculated fluid flow velocity vector 638. However, the direction of theobserved motions on the imaging plane due to refraction can beconsistent over time as the fluid changes its position. According toembodiments of the invention, the observed motions can be tracked overtime to recover a projected 2-D motion of the refractive fluid 628.

To obtain a calculated fluid flow such as the calculated fluid flowvelocity vector 638 shown in FIG. 6B, the observed scene can be modeledas composed of a stationary background, Ĩ(x, y) (such as the texturedbackground 426), and a refractive field, r(x, y, t), (such as therefractive fluid 628) translating in between the scene and the camera:

I(x+r _(x)(x, y, t), y+r _(y)(x, y, t)={tilde over (I)}(x, y),  (3)

The refractive field, r, indicates the spatial offset in the imagingplane, caused by light rays bending due to index of refraction gradientsbetween the scene and the camera. For example, such refractive gradientscan be due to bubbles of hot air traveling with the velocity of movingair.

The inventors have recognized that it can be further assumed that themoving refraction gradients imaged to a point (x, y) have a singlevelocity, u(x, y). In other words, the apparent motions v(x,y) areconsidered to be constant from one video frame to another frame. Underthese conditions, namely when motions, or translation, of the refractivefield is assumed to be the exclusive cause of local image velocitychanges, this “refraction constancy” can be exploited to explain theobserved local velocity changes by the translation, u(x, y), of anassumed refractive field. In other words, the frame-to-frame motions ina video are considered to be a function of motion of the refractivefield r(x,y,t). This yields a “refraction constancy” equation:

r(x, y, t)=r(x+u _(x) Δt, y+u _(y) Δt, t+Δt).  (4)

It is shown hereinafter in the section labeled “Proof of v=δr/δt” that,under the condition of Equation 4, the temporal derivatives of therefraction field correspond directly to the observed motion in thesequence. That is, v=δr/δt. Additionally taking the partial derivativeof Equation 4 with respect to t, the observed motion can be written as

v(x, y, t)=v(x+u _(x) Δt, y+u _(y) Δt, t+Δt).  (5)

Since, the v(x,y,t) constitute frame-to-frame representations of motionsin a video, and since r(x,y,t) in Equation 4 is expressed as a functionof motion u(x,y) of the refractive field, an implementation of Equation5 by a processor is, thus, one way to correlate, over time,representations of motions in the video from frame to frame as afunction of motion of the refractive field.

It should be pointed out that the “correlating” performed in embodimentsof the present invention should be distinguished from the “correlating”performed in signal processing. Further, while one way of correlatingrepresentations of motions in the video as a function of motion of arefractive field is described above, the embodiments can include otherways of correlating. For example, another way to correlaterepresentations of motions in the video is to first filter therepresentations of motion by complex oriented filters, and then extractthe phase change of filters response between frames.

Assuming v varies smoothly and that Δt is small, a first order Taylorexpansion can again be used to approximate Equation 5 as

$\begin{matrix}{{{\frac{\partial v}{\partial x}u_{x}} + {\frac{\partial v}{\partial y}u_{y}} + \frac{\partial v}{\partial t}} = 0.} & (6)\end{matrix}$

Equation 6 can be used to develop and implement a two-step approach forsolving for the fluid motion, u. The observed or apparent motions, v,can first be calculated using the standard optical flow equation(Equation 2). Then, optical flow can be applied again, this time on theestimated motions, to get the actual motion of the refractive fluid.Solving for u(x,y) for at least one point (x,y), therefore, constitutescalculating a velocity of the refractive field based upon the correlatedrepresentations v(x,y,t) of the observed motions. Further, solving forseveral points can be carried out to produce a fluid velocity vectorfield. Since the observed motions correspond directly to changes in therefraction field, and following the refraction constancy assumption,computing the optical flow of the optical flow will yield actual orrefractive fluid flow.

Refractive Flow Versus Optical Flow

Applying standard optical flow to an observed video sequence will notyield the correct motions of the refractive fluid. To illustrate this,the refraction constancy equation (Equation 4) may be considered again.Approximating Equation 4 by the first order using a Taylor expansionyields

$\begin{matrix}{{{\frac{\partial r}{\partial x}v_{x}} + {\frac{\partial r}{\partial y}v_{y}} + \frac{\partial r}{\partial t}} = 0.} & (7)\end{matrix}$

Note that the optical flow result is equal to the change of refractionfield, v=δr/δt. Then from Equation 7, it can be observed that:

$\begin{matrix}{{v = {Du}},{{{where}\mspace{14mu} D} = \begin{bmatrix}\frac{\partial r_{x}}{\partial x} & \frac{\partial r_{x}}{\partial y} \\\frac{\partial r_{y}}{\partial x} & \frac{\partial r_{y}}{\partial y}\end{bmatrix}},{u = \begin{bmatrix}u_{x} \\u_{y}\end{bmatrix}},} & (8)\end{matrix}$

where the matrix D in Equation 8 can be called the refraction Jacobianmatrix (or the refraction matrix).

Equation 8 shows that the observed motions, v, are in fact the productof the refractive matrix D and the fluid flow v. This means that, underthe image formation model, the observed motions are caused by bothrefraction gradients and fluid motion. For example, the same observedmotion can result from large refraction changes and small fluid motion,or large fluid motion and small refraction changes.

The form of Equation 8 provides a few insights on refractive flow.First, it can be noted that refractive flow can be obtained where flowis not completely stable laminar flow, in which case the refractionmatrix, D, would be zero. This is analogous to the aperture problemknown for optical flow. Second, it can be noted that optical flow(brightness constancy) alone will not be able to disambiguate fluidmotion from refraction changes. In fact, with the refraction constancyassumption (Equation 4), these two types of motions are implicitlydisambiguated by attributing lower-spatial-frequency motion to fluidmotion, and higher-spacial-frequency motion to refraction gradients.

Refractive Flow based on Phase Analysis

It is useful to note that the approach to fluid flow described aboveprovides no restriction on how to actually compute optical flow in arefractive flow methods. For example, good results can be obtained whenusing local phase changes for initially characterizing the observed orapparent motions, instead of using optical flow on the intensities. Itshould be noted that motions due to refraction can be small, even on theorder of one pixel per frame or less.

Phase changes can be obtained by building a two-orientation, two-levelcomplex steerable pyramid and applying DC-balanced temporal high passfilter to the phases (e.g., convolving with [1, −1]). This allows one toavoid having to use an undistorted reference frame, as done inconventional background-oriented schlieren techniques. Thisdecomposition provides both amplitude and phase, which, respectively,capture the texture strength and motion.

A temporal Gaussian blur can be applied to an input video sequence, andan amplitude-weighted spatial blur can be applied to the phases. Thesevariations improve the signal-to-noise ratio of the signalsignificantly, allowing extraction of very subtle signals due torefraction. To deal with small camera motions, the amplitude weightedmean of the phase over a single orientation, scale, and frame can besubtracted from a phase signal.

After extracting and processing the phases, feature vectors can becreated by combining the phase from 19 frames in time (the frame ofinterest itself, 9 frames before, and 9 frames after the frame ofinterest). Using a wider temporal extent to compute features can providea richer representation and can improve results relative to onlytracking features between single frames.

Notably, even ordinary consumer cameras can be used to acquire videoimages in embodiments of the invention. For example, the candle, wind,and landing sequences shown in FIGS. 1A and 1B were taken with ordinaryconsumer DSLR cameras. However, higher performance cameras may also beused in some applications for improved results. For example, ahigh-speed camera was used to film the takeoff video images shown inFIG. 1B. Specifically, the takeoff sequence was filmed at 500 frames persecond.

Where a background is sufficiently textured, even small differences inair temperatures can be visualized with devices implementing the examplemethods described above. For example, in a separate test video sequence(not shown), the flow of heated air around a person was visualized.Since the temperature difference between a person and the ambient aircan be very small, the amount of refraction produced by heated airaround person can likewise be small, resulting in deflections as smallas 100th of a pixel. In this test case, the signal-to-noise ratio wasincreased by having the person running up and down 20 flights of stairsand then acquiring the video of the person in a 4° C. refrigerator usinga well-textured background consisting of a multiscale wavelet noisepattern.

However, embodiments of the invention can also visualize scenes withoutan artificial background like the multiscale wavelet noise patternpreviously mentioned. In the wind, take off, and landing videos, forexample, atmospheric wind patterns are visualized using the forest andbuildings as the textured background to visualize changes in index ofrefraction at a given position in the scene.

It should be noted that measuring wind speed is an important factor inweather forecasting. Embodiments of the invention allow for densersampling of wind speed than the point measurements given by ananemometer. Thus, embodiments of the invention can improve the accuracyand range of forecasts.

Proof of v=δr/δt

If the background is considered to be stationary, then the brightnessconstancy assumption can be applied to fluid objects:

I(x+r _(x)(x, y, t), y+r _(y)(x, y, t), t)=I(x+r _(x)(x, y, t+Δt), y+r_(y)(x, y, t+Δt), t+Δt).  (9)

A first order approximation to the right hand side of Equation 9 yields

${{I\left( {{x + {r_{x}\left( {x,y,t} \right)}},{y + {r_{y}\left( {x,y,t} \right)}},t} \right)} = {{I\left( {{x + {r_{x}\left( {x,y,t} \right)}},{y + {r_{y}\left( {x,y,t} \right)}},{t + {\Delta \; t}}} \right)} + {\frac{\partial I}{\partial x}\frac{\partial r_{x}}{\partial t}\Delta \; t} + {\frac{\partial I}{\partial x}\frac{\partial r_{y}}{\partial t}\Delta \; t}}},$

which yields

$\begin{matrix}{{{\frac{\partial I}{\partial x}\frac{\partial r_{x}}{\partial t}} + {\frac{\partial I}{\partial y}\frac{\partial r_{y}}{\partial t}} + \frac{\partial I}{\partial t}} = 0.} & (10)\end{matrix}$

Therefore, the observed image motion v is equal to the temporal gradientof the refractive field δr/δt.

Conditions for 2D Refraction Constancy

In this section, it is shown when the 2D refraction constancy assumptionholds. In general, a fluid element is described by aspatiotemporally-varying index of refraction n(x, y, z, t). The 2Drefraction field is determined by the gradient of the index ofrefraction. It can be assumed that index of refraction does not changewhen a fluid element moves (3D refraction constancy). That is,

n(x, y, z, t)=n(x+Δx, y+Δy, z+Δz, t+Δt).

Suppose a ray traveling in this field is defined as f(s) =(f_(x)(s),f_(y)(s), f_(z)(s)). When the index of refraction is close to 1 (as inair), the following equation holds

$\begin{matrix}{{\frac{dd}{ds} = {\nabla n}},} & (11)\end{matrix}$

where

$d = {\frac{d\overset{->}{r}}{ds}/{\frac{d\overset{->}{r}}{ds}}}$

is a unit vector describing the local ray direction.

If the distance between the air flow and background is much larger thanthe thickness of the air, then the refractive field is approximatelyequal to

$\begin{matrix}{{r\left( {x,y,t} \right)} = {{- {hP}}{\int_{{f{(s)}}\text{:}\mspace{14mu} C_{x,y,t}}{{\nabla{n\left( {{f(s)},t} \right)}}{ds}}}}} & (12)\end{matrix}$

where h is the distance between the air and the background, P is aprojection matrix that projects the motion in 3D to 2D, and C_(x, y, t)is the ray that hits the point (x, y) on the image plane at time t. Itcan be assumed that all the points on the ray path C_(x, y, t) share thesame motion.

When the motion of air is small, the ray hits (x, y) on the image planeat time t passes through the similar region as the ray hits (x+v_(x)Δt,y+v_(y)Δt) on the image plane at time t+Δt. Therefore,

∫_(f(s):  C_(x, y, t))∇n(f(s), t)ds = ∫_(f(s):  C_(x + v_(x)Δ t, y + v_(y)Δ t, t + Δ t))∇n(f(s), t + Δ t)ds,

Furthermore, if it is further assumed that over a short time, the motionalong z-dimension is negligible compared with the distance between theair and the background, then h in Equation 12 can be considered to beconstant. Therefore, r(x, y, t)=r(x+v_(x)Δt, y+v_(y)Δt, t+Δt). Thisproves the 2D refraction constancy.

Calculations have been derived for tracking the movement of refractivefluids in a single video and for recovering a 3-D position of points ona fluid surface from stereo video sequences. Both of these calculationsare based on the refractive constancy assumption previously described,namely that intensity variations over time (the wiggles or apparentmotions) can be explained by the motion of a constant refraction field.

Calculations for Three Dimensional (3D) Measurements and Uncertainty

FIG. 6C includes a diagram 600 c illustrating measurement of a depth630D from stereo video images in accordance with embodiments of theinvention. Similar to FIG. 6B, apparent motion velocity vectors at acamera imaging plane are viewed as the refractive fluid moves from atime t to time t+Δt plus Delta T such that apparent motion velocityvectors, or wiggles, can be observed in two different imaging planes oftwo different cameras and can be matched or correlated over space torecover the depth of the fluid.

FIG. 6C is similar to FIG. 6B in that apparent motion velocity vectors,or wiggles, are viewed using light rays passing from a texturedbackground 426 through a refractive fluid 628 that is in motion.However, FIG. 6C includes two cameras, a left camera with a left cameraimaging plane 670 _(L) and a right camera with the right camera imagingplane 670 _(R). Left camera has a left center of projection 690 _(L) andobserves wiggles 614 _(L). The right camera has a right center ofprojection 690 _(R) and sees wiggles 614 _(R) as the refractive fluid628 moves during the small time Δt.

The wiggle features in an input video are computed using optical flow,and then using those features to estimate the motion and depth of thefluid, by matching them across frames and viewpoints.

The distortion of the background caused by the refraction is typicallyvery small (in the order of 0.1 pixel) and therefore often hard todetect. The motion features have to be extracted carefully to overcomeinherent noise in the video, and to properly deal with regions in thebackground that are not sufficiently textured regions, in which theextracted motions are less reliable. To address these issues,probabilistic refractive flow and stereo methods that maintain estimatesof the uncertainty in the optical flow, the refractive flow, and thefluid depth are presented hereinafter.

The proposed methods have several advantages over existing methods: (1)a simple apparatus that can be used outdoors or indoors, (2) they can beused to visualize and measure air flow and 3D location directly fromregular videos, and (3) they maintain estimates of uncertainty.

Gradients in the refractive index of air can show up as minute motionsin videos. Theorems are stated hereinafter that establish the relationbetween those observed motions in one or more cameras and the motion anddepth of refractive objects in a visual scene.

In the monocular case previously shown in FIG. 6B, a video cameraobserves a static background 426 through a single refractive fluid layer628. When the fluid is moving, the same point on the background maytrace back to different points on the image plane due to refractionchanges (see the dashed and solid lines in FIG. 6B. Such observedmotions can be called “refraction wiggles”, or “wiggles” for short. Whena fluid object with a non-uniform surface moves a small amount from timet₁ (solid shape) to time t₁+Δt (dashed shape), the light ray passingthrough a point B on the background changes due to the refraction (fromsolid line to dashed line), producing a visual shift on the image plane(short arrow), which is the wiggle and can be represented by theapparent motion velocity vectors 614 b ₁ and 614 b ₂. At time t₂, thefluid moves to another location, and the projection of the movement ofthe fluid onto the image plane (long black arrow) can be recovered. Thedirection of the wiggle and the direction of the refractive flow can bedifferent, as the wiggle direction also depends on the shape of therefractive element.

Because the input in FIG. 6B is only a monocular video sequence for onecamera imaging plane 670, the method can at most recover a 2D projectionof the motion of the refractive fluid 628.

The task of refractive flow is to recover the projected 2D fluid motionfrom an input video sequence.

Similarly, in the stereo case, two cameras with image planes 690 _(L)and 690 _(R) image static background 426 through a refractive fluidlayer 628. If a standard stereo matching directly on the input stereosequence is applied, the method will recover the depth of (solid) pointsin the background. In contrast, in refractive stereo it is desirable torecover the depth of the (transparent) fluid layer, stereo fusing on themotion wiggles rather than the image intensities.

Methods herein after presented for both refractive flow and refractivestereo are based on this observation: for a point on the fluid object,its refraction wiggle is constant over a short time and across differentviews. To state this observation more formally, first the wiggle that apoint on a fluid layer generates can be defined.

Definition 1 (Refraction wiggle): Let A be a point on the refractivefluid layer, B be the intersection between the background and the lightray passing through A and the center of projection at time t, and Δt bea short time interval. Then the wiggle of A from time t to t+Δt is theshift of the projection of B on the image plane during this time.

Then this definition leads to the following two refractive constancytheorems.

Theorem 1 (Refractive flow constancy): Suppose the fluid object does notchange its shape and index of refraction during a short during [t1, t2].Then for any point on the fluid object, its wiggle v(t1) at t1 equals toits wiggle v(t2) at t2.

Theorem 2 (Refractive stereo constancy): Suppose there are at least two(n≥2) cameras imaging a refractive fluid object, and they are allparallel and close to each other. Then at any time t, and for any pointon the fluid object, the corresponding wiggle in all the cameras is thesame.

Proofs of Theorems 1 and 2 are included hereinafter in the SupplementaryMaterial section.

FIGS. 6B and 6C illustrate these two theorems, respectively. In themonocular case shown in FIG. 6B, when the fluid moves from time t₁ tot₁+Δt₁ and from time t₂ to t₂+Δt₂, it generates wiggles shown by therespective arrows, and according to Theorem 1, these two wiggles areequal. Similarly, in the stereo case in FIG. 6C, when the fluid movesfrom t to t+Δt, it generates a wiggle on the left and right views (shownby the respective arrows 614C_(L) and 614C_(R)), and according toTheorem 2, these two wiggles are also equal.

The practical implication of these theorems is that matching theprojection of a point on the fluid object across frames (over time) orviewpoints (over space) can be done by matching the observed wigglefeatures. That is, while the position of intensity texture features isunrelated to the fluid 3D structures, the position of wiggle featuresrespect features on the fluid surface and can serve as an input featurefor optical flow and stereo matching methods. Hereinafter, derivationsof optical flow and stereo methods for tracking and localizingrefractive fluids are presented.

The goal of fluid motion estimation is to recover the projected 2Dvelocity, u(x, y, t), of a refractive fluid object from an observedimage sequence, I(x, y, t). As described in the previous section, thewiggle features v(x, y, t), not the image intensities I(x, y, t), movewith the refractive object. Thus, estimating the fluid's motion can bedone using two steps: 1) computing the wiggle features v(x, y, t) froman input image sequence I(x, y, t), and 2) estimating the fluid motionu(x, y, t) from the wiggle features v(x, y, t). Each of these steps isdescribed in turn below.

Computing Wiggle Features

The brightness constancy assumption in optical flow is that any changesin pixel intensities, I, are assumed to be caused by a translation withmotion v=(v_(x), v_(y)) over spatial horizontal or vertical positions, xand y, of the image intensities, where v_(x) and v_(y) are the x and ycomponents of the velocity, respectively. That is,

I(x, y, t+dt)=I(x+v _(x) dt, y+v _(y) dt, t).

Based on this brightness constancy equation, a traditional way tocalculate the motion vector v is to minimize the following optical flowequation:

$\overset{\sim}{v} = {{\arg {\min\limits_{u}{\sum\limits_{x}{\alpha_{1}{{{\frac{\partial I}{\partial x}v_{x}} + {\frac{\partial I}{\partial y}v_{y}} + \frac{\partial I}{\partial t}}}^{2}}}}} + {\alpha_{2}\left( {{\frac{\partial v}{\partial x}}^{2} + {\frac{\partial v}{\partial y}}^{2}} \right)}}$

where α₁ and α₂ are weights for the data and smoothness terms,respectively.

Estimating Fluid Motion

Let u_(x) and u_(y) be the x and y components of the fluid's velocity.The refractive constancy equation for a single view sequence is then:

v(x, y, t+Δt)=v(x+u _(x) Δt, y+u _(y) Δt, t)

Notice that refractive constancy has the exact same form as brightnessconstancy (Equation 1), except that the features are the wiggles vrather than the image intensities I. Applying optical flow to the wigglefeatures v (rather than to the intensities, I), will yield the fluidmotion u. The fluid motion u by minimizing the (fluid flow) followingequation:

$\overset{\sim}{u} = {{\arg {\min\limits_{u}{\sum\limits_{x}{\beta_{1}{{{\frac{\partial\overset{\sim}{v}}{\partial x}u_{x}} + {\frac{\partial\overset{\sim}{v}}{\partial y}u_{y}} + \frac{\partial\overset{\sim}{v}}{\partial t}}}^{2}}}}} + {\beta_{2}\left( {{\frac{\partial u}{\partial x}}^{2} + {\frac{\partial u}{\partial y}}^{2}} \right)}}$

This is similar to the Horn-Shark optical flow formulation and similarto optical flow methods, and a multi-scale iterative method can be usedto solve it.

FIGS. 1D(a)-(d) demonstrate a result of this refractive flow method,when applied to a video of a burning candle (candle). First, wigglefeatures (b₁) and (b₂) are extracted from the input video (a). Sincewiggle features move coherently with the air flow, the method correctlyestimates the motion of the thermal plume rising from the candle. Noticethat the observed motions (wiggles) have arbitrary directions, yet theestimated refractive flow (c) is much more coherent.

Such processing is very sensitive to noise, however, as can be seen inFIG. 1D(b₁) and (b₂). The problem can be even more severe for lesstextured backgrounds. This motivates the probabilistic formulationbelow.

Both the refractive flow, and its uncertainty can be estimated. Considera background that is smooth in the x direction and textured in theydirection. Due to the aperture problem known in the art of optical flow,the flow in the x direction may be dominated by noise, while the opticalflow in they direction can be clean. Knowing the uncertainty in the flowallows uncertain estimates to be down-weighted, increasing therobustness of the method.

To find the variance of the optical flow, the following equation can beformulated as a posterior distribution:

${P\left( v \middle| I \right)} = {\exp \left( {{- {\sum\limits_{x}{\alpha_{1}{{{\frac{\partial I}{\partial x}v_{x}} + {\frac{\partial I}{\partial y}v_{y}} + \frac{\partial I}{\partial t}}}^{2}}}} + {\alpha_{2}{\frac{\partial v}{\partial x}}^{2}} + {\alpha_{2}{\frac{\partial v}{\partial y}}^{2}}} \right)}$

Here, P(v|I) is a Gaussian distribution, and the mean of P(v|I) equalsto the solution of the original optical flow equation for {tilde over(v)} given above. With this formulation, the variance of the opticalflow (the wiggle features) can also be calculated. See the supplementarymaterial for the detailed calculation. Let {tilde over (v)} and Σv bethe mean and covariance, respectively, of the wiggle features computedfrom Equation the equation above for posterior distribution P(v|I). TheL2-norm for regularization is used, in contrast to robust penaltyfunctions such as L1 norm traditionally used by optical flow methods,since fluid objects, especially hot air or gas, do not have the clearand sharp boundaries of solid objects.

Then, with the variance of the wiggle features, we can reweight thefluid flow equation can be reweighted as follows:

$\overset{\sim}{u} = {{\arg {\min\limits_{u}{\sum\limits_{x}{\beta_{1}{{{\frac{\partial\overset{\sim}{v}}{\partial x}u_{x}} + {\frac{\partial\overset{\sim}{v}}{\partial y}u_{y}} + \frac{\partial\overset{\sim}{v}}{\partial t}}}_{\Sigma_{v}}^{2}}}}} + {\beta_{2}\left( {{\frac{\partial u}{\partial x}}^{2} + {\frac{\partial u}{\partial y}}^{2}} \right)} + {\beta_{3}{u}^{2}}}$

Here the squared Mahalanobis distance is denoted as ∥x∥_(Σ) ²=x^(T)Σ⁻¹x.In this formulation, the data term is reweighted by the variance of theoptical flow to robustly estimate the fluid motion: wiggle features withless certainty, such as motions measured in regions of low-contrast, orof flat or 1-dimensional structure, will have low weight in fluid flowequation. To increase the robustness, the magnitude of u can also be toavoid estimating spurious large flows. The inclusion of the aboveuncertainty information leads to more accurate estimation of the fluidmotion, as shown in FIG. 1D(d).

In practice, calculating the covariance matrix precisely for each pixelis computationally intractable, as the marginal probability distributionfor each optical flow vector needs to be computed. To avoid thiscalculation, we concatenate all the optical flow vectors can beconcatenated into a single vector and its covariance can be computed.See the supplementary material given hereinafter for the details. Also,the fluid flow equation still has a quadratic form, so the posteriordistribution of the fluid flow u can be modelled as a Gaussiandistribution, and its variance can be computed. This variance serves asa confidence measure in the estimated fluid motion.

The goal of fluid depth estimation is to recover the depth D(x, y) of arefractive fluid object from a stereo sequence I_(L)(x, y, t) andI_(R)(x, y, t) (FIG. 2(b)). According to the refractive stereo constancytheorem (Theorem 2), the depth of the fluid object can be found bystereo-matching the refraction wiggles from the left and right views. Sofirst the method discussed in the previous section can be used tocalculate the mean and variance of the optical flows in the left andright views, v_(L)˜N({tilde over (v)}_(L), Σ_(L)) and v_(R)˜N({tildeover (v)}_(R), Σ_(R)), respectively.

A discrete Markov Random Field (MRF), common for stereo matching canthen be used to regularize the depth estimates. Formally, let x_(L), andx_(R) be the projection of a point on the fluid object onto the left andright image plane, respectively, and define ECCV-14 submission ID 1550 9disparity as d=x_(L)−x_(R). The disparity map can be solved first byminimizing the following objective function:

$\overset{\sim}{d} = {{\min\limits_{d}{\sum\limits_{x,y}{f\left( {{v_{R}\left( {x,y} \right)},{v_{L}\left( {{x + {d\left( {x,y} \right)}},y} \right)}} \right)}}} + {\alpha \left( {{\frac{\partial d}{\partial x}}^{2} + {\frac{\partial d}{\partial y}}^{2}} \right)}}$

where f(v_(R), v_(L)) is the data term based on the observed wigglesv_(R) and v_(L), and the last two terms regularize the disparity field.The inventors have found that using the L₂ norm for regularizationgenerates better results overall, better explaining the fuzzy boundariesof fluid refractive objects.

As with the refractive flow, the data term can be weighted by thevariance of the optical flow to make the depth estimation robust topoints in a scene where the extracted wiggles are not as reliable. Toachieve this, the data term, f(v_(R), v_(L)), can be defined as the logof the covariance between the two optical flows from the left and rightviews:

f({tilde over (v)} _(R) , {tilde over (v)} _(L))=log cov(v _(R) , v_(L))=½log|Σ_(L)+Σ_(r)|+½∥v _(R) −v _(L)∥_(Σ) _(L) _(+Σ) _(R) ²

where ∥v _(R) −v _(L)∥_(Σ) _(L) _(+Σ) _(R) ²=(v _(R)−v_(L))^(T)(Σ_(L)+Σ_(R))⁻¹(v _(R)−v _(L)). This data term will assign ahigher penalty to inconsistencies in the wiggle matching where thewiggles are more reliable, and a lower penalty where the wiggles areless reliable (typically where the background is less textured and theoptical flow is noisy). The choice of the log of the covariance as thedata term is not arbitrary, and it is the log of the conditionalmarginal distribution of v_(L) and v_(R), given that v_(L) and v_(R)match. See the supplementary material given hereinafter for a detailedderivation.

With calibrated cameras, the depth map, D(x, y), can be computed fromthe disparity map, d(x, y).

Qualitative Results

FIG. 7 shows several results of refractive flow analysis using a singlecamera.

FIG. 7 contains several examples of measuring and localizing hot airradiating from several heat sources, such as people, fans, and heatingvents. Refer to the supplementary materials given hereinafter for thefull sequences and results.

All the videos were recorded in raw format to avoid compressionartifacts. To deal with small camera motions or background motions, themean flow for each frame can be subtracted from the optical flow result.For each sequence captured, a temporal Gaussian blur was first appliedto the input sequence in order to increase the SNR. The high-speedvideos were captured using a high-speed camera. For some of the indoorhigh-speed sequences, a temporal band-stop filter was used to removeintensity variations from the light intensities modulated by AC power.

The refractive flow method was first tested in controlled settings usinga textured background. In the “hand” column, a video at 30 frames persecond (fps) was taken of a person's hand right after holding a cup ofhot water.

Heat radiating upward from the hand was recovered. In hairdryer, a 1000fps high-speed video of two hairdryers placed opposite to each other(the two dark shapes in the top left and bottom right are the front endsof the hairdryers) was taken, and two opposite streams of hot air flowswere detected.

The “kettle” and “vents” columns demonstrate the result on more naturalbackgrounds. In “vents” (700 fps), the background is very challengingfor traditional background oriented schlieren (BOS) methods, as someparts of the background are very smooth or contain edges in onedirection, such as the sky, the top of the dome, and the boundary of thebuildings or chimneys. BOS methods rely on the motion calculated frominput videos, similar to the wiggle features shown in the second row ofFIG. 7, which is very noisy due to the insufficiently texturedbackground. In contrast, the fluid flow (third row of FIG. 7) clearshows several flows of hot air coming out from heating vents. Bymodeling the variance of wiggles in the probabilistic refractive flow,most of the noise in the motion is suppressed. The bottom (fourth) rowof FIG. 7 shows the variance of the estimated refractive flow (thesquare root of the determinant of the covariance matrix for each pixel).

Quantitative Evaluation

FIGS. 8A-8E illustrate quantitative evaluation of the refractive flowmethod disclosed herein. Simulated fluid density (FIG. 8A) and velocityfield (FIG. 8B) were generated by Stable Fluid, a physics-based fluidflow simulation technique, and rendered on top of three differenttextures (FIG. 8D). The recovered velocity field from one of thesimulations in which the fluid was at 320 Celsius (FIG. 8C) is similarto the ground truth velocity field (FIG. 8B). Quantitative evaluation isgiven in (FIG. 8E). As expected, larger temperature-related index ofrefraction differences between the moving fluid and the background givebetter flow estimates.

To quantitatively evaluate the fluid velocity recovered by the disclosedrefractive flow method, it on was also tested simulated sequences withprecise ground truth reference. A set of realistic simulations ofdynamic refracting fluid was generated using Stable Fluids, aphysics-based fluid flows simulation technique, resulting in fluiddensities and (2D) ground truth velocities at each pixel over time, asillustrated in FIGS. 8A-8B. The simulated densities were then used torender refraction patterns over several types of background textureswith varying spatial smoothness (FIG. 8D). To render realisticrefractions, it was assumed that the simulated flow was at a constantdistance between the camera and background with thickness dependinglinearly on its density. Given a scene temperature (20° Celsius wasused) and a temperature of the fluid, the amount of refraction at everypoint in the image can be computed. The refractive flow methodpreviously described was then applied to the refraction sequences. Theendpoint error of the recovered fluid motion at different temperature isshown in (FIG. 8E). All the sequences and results are available in thesupplemental material provided hereinafter.

FIGS. 9A-9D illustrate a controlled experiment to evaluate thevelocities estimated using a refractive flow method. (FIG. 9A) Theexperiment setup. (FIG. 9B) A representative frame from the capturedvideo. (FIG. 9C) The mean velocity of the hot air blown by thehairdryer, as computed by our method, in meters per second (m/s). (FIG.9D) Numerical comparison of our estimated velocities with velocitiesmeasured using a velometer, for the four points marked x₁-x₄ in (FIG.9C).

To demonstrate further that the magnitude of the motion computed by therefractive flow method is correct, a controlled experiment as shown inFIGS. 9A-9D. A velocimeter was used to measure the velocity of hot airblown from a hairdryer, the velocities were compared with the velocitiesextracted by the refractive flow method. To convert the velocity on theimage plane to the velocity in the real world, the hairdryer was setparallel to the imaging plane and a ruler was placed on the same planeof the hairdryer. The velocity of the flow was measured at fourdifferent locations as shown in FIGS. 9C-9D. Although the estimated flowdoes not match the velocimeter flow exactly, they are highly correlated(FIG. 9D) and the agreement is believed to be accurate to within theexperimental uncertainties.

Qualitative Results

FIG. 10 illustrates fluid depth recovery from stereo sequences. Thefirst and second rows show the input videos overlayed by wiggle features(optical flow) calculated. third line shows the disparity map (weightedby the confidence of disparity) of the fluid object.

Several stereo sequences were captured using two cameras at acquiringimages at 50 FPS. The two cameras were synchronized via genlock and aglobal shutter was used to avoid temporal misalignment. All videos werecaptured in 16 bit grayscale.

A controlled experiment to evaluate the velocities estimated by themethod. (a) The experiment setup. (b) A representative frame from thecaptured video. (c) The mean velocity of the hot air blown by thehairdryer, as computed using the method, in meters per second (m/s). (d)Numerical comparison of our estimated velocities with velocitiesmeasured using a velometer, for the four points marked x₁-x₄ in (c).

The third row of FIG. 10 shows the disparity map of air flow. In thecandle column are shown two plumes of hot air at different depthrecovered. In lighting, three lights were positioned at differentdistances from the camera: the left one was the closest to the cameraand the middle one was the furthest. The disparities of three plumescalculated in accordance with the method match the locations of thelights. In monitor, the method recovers the location of hot air risingfrom the top of a monitor. The right side of the monitor is closer tothe camera, and so the hot air on the right has larger disparity. In thesupplementary material given hereinafter are shown the recoveredpositions of the hot air as the heater rotates back and forth over time.

Quantitative Evaluation

A synthetic simulation was performed to demonstrate that probabilisticrefractive stereo is robust to less-textured backgrounds. The simulationsetup is similar to one used for the refractive flow method, and itsdetails and the results are available in the supplementary materialgiven hereinafter. The probabilistic framework allows for depthestimates of the flow even over image regions of only 1-dimensionalstructure.

Supplementary Material Refractive Stereo: Quantitative Evaluation

Refractive stereo results were evaluated quantitatively in two ways.First, for natural stereo sequences, recovered depths of the refractivefluid layers were compared with those of the heat sources generatingthem, as computed using an existing stereo method (since the depth ofthe actual heat sources, being solid objects, can be estimated wellusing existing stereo techniques). More specifically, a region on theheat source was chosen and another region of hot air right above theheat source was chosen, and the average disparities in these two regionswas compared. Experiments showed that the recovered depth map of the(refractive) hot air matches well the recovered depth map of the (solid)heat source, with an average error of less than a few pixels.

Second, the refractive stereo method was evaluated on simulatedsequences with ground truth disparity. The simulation setup is similarto one used for the refractive flow method, except that the ground truthdisparity map was manually specified as shown in FIGS. 11A-11B.

To evaluate the performance of the method, four different backgroundpatterns were generated as shown in FIGS. 11A-11B: 1) weakly textured inboth x and y directions, 2) strongly textured in x direction, 3)strongly textured in y direction, 4) strongly textured in bothdirections. All combinations of these backgrounds were generated in theleft and right views to yield 4×4=16 stereo sequences. For comparison, arefractive stereo method that does not consider the uncertainty in theoptical flow was also implemented. FIGS. 11A-11B show the disparitiesrecovered by both refractive stereo (without uncertainty; the baseline)and probabilistic refractive stereo (with uncertainty), together withtheir corresponding root-mean-square error (RMSE). The simple refractivestereo worked well when the backgrounds of both the left and right viewswere strongly textured in both the x and y directions. When thebackground was weakly textured in one direction, the optical flow inthat direction was noisy and thus the recovered disparity would besignificantly less accurate.

The probabilistic refractive stereo method was able to handle weakertextures. As long as one direction of the background was textured inboth views, the disparity map was accurately recovered. For example, inthe second row and second column in FIG. 1B, the optical flow in the xdirection is noisy, while the optical flow in they direction is clean,because the background is very smooth in x direction. In theprobabilistic refractive stereo, the optical flow results were weightedby how accurately they are estimated when calculating the disparity.Therefore, the optical flow in the x direction would have a smallereffect on the final result than the optical flow in they direction, andthe disparity map would be calculated correctly. This demonstrates therobustness of the method to partially textured backgrounds.

FIG. 12 is similar to FIG. 6B but also illustrates quantities relevantto a proof of refractive constancy. As described above, the refractivewiggle is defined as:

Definition 1 (Refraction wiggle) Let x′ be a point on the refractivefluid layer, x″ be the intersection between the background and the lightray passing through x′ and the center of projection at time t, and Δt bea short time interval. Then the wiggle of x′ from time t to t+Δt is theshift of the projection of x″ on the image plane during this time.

This definition leads to the following two refractive constancytheorems.

Theorem 1 (Refractive flow constancy) Suppose the fluid object does notchange its shape and index of refraction during a short time interval[t₁, t₂]. Then for any point on the fluid object, its wiggle v(t₁) at t₁equals its wiggle v(t₂) at t₂.

Proof Let x′_(t) ₁ be the location of a point on the fluid object attime t₁, and x′_(t) ₂ be the location of the same point at time t₂. Itwill be shown that the wiggles at these two points and times are thesame.

Because wiggles are defined by shifts in the image plane, rays are firsttraced to determine what points in the image plane correspond to theselocations on the fluid object. At time ti, i=1, 2, an undistorted ray isemitted from the center of the projection to point x′_(t) _(i) with theangle α′_(t) _(i) with respect to vertical (solid lines in FIG. 2). Thisray is bent by refraction as it passes through the fluid and travels atan angle α″ _(t) _(i) with respect to the vertical before finallyintersecting the background at x″_(t) _(i) .

At a successive time t_(i)+Δt, the fluid object moves to a new location(dashed gray blob in FIG. 12). The background point x″_(t) _(i) thencorresponds to a different point x′_(t) _(i) +Δt on the fluid object anda different point x_(t) _(i) +Δt on the image plane. The ray from thebackground to the fluid object now makes an angle of α′_(t) _(i) +Δtwith the vertical and the same ray as it travels from the fluid objectto the center of projection makes an angle of α′_(t) _(i) +Δt with thevertical. The wiggle at time t_(i) is the distance between x_(t) _(i)and x_(t) _(i) +Δt, denoted as {right arrow over (x_(t) ₁ x_(t) ₁ +Δt)}(blue and red arrows in FIG. 12). It can be proven that the wiggles attime t₁ and t₂ are equal to each other. That is,

{right arrow over (x _(t) ₁ x _(t) ₁ +Δt)}={right arrow over (x_(t) ₂ x_(t) ₂ +Δt)}

To prove this, it will first be shown that

$\overset{x_{t_{1} + {\Delta \; t}}^{\prime}}{\rightarrow}\mspace{14mu} {{and}\mspace{14mu} \overset{x_{t_{2} + {\Delta \; t}}^{\prime}}{\rightarrow}}$

same and are the same point on the fluid object, or equivalently, theshifts x′_(t) _(i) x′_(t) ₁ +Δt and x′_(t) ₂ x′_(t) ₂ +Δt are equal. Bythe definition of wiggle, it is known that x′_(t) ₁ and x′_(t) ₁ referto the same point on the refractive object. It can be proven that x′_(t)₁ +Δt and x″_(t) ₂ +Δt are equal.

Let z, z′, and z″, respectively, be the depths of the image plane, thefluid layer, and the background from the center of projection (FIG. 2).To find the relationship between x′_(t) ₁ +Δt and x′_(t) ₂ +Δt, x′_(t) ₂+Δt is solved for in terms of x′_(t) ₁ , x′_(t) ₂ , and x′_(t) ₁ +Δt. Asimple geometric calculation shows (assuming that all the incident angleα′t_(i) is small, so that a′_(t) _(i) ≈sin(α′_(t) _(i) )):

${\frac{x_{t_{i}}^{\prime} - o}{z^{\prime}} = {\alpha_{t_{i}}^{\prime}\mspace{31mu} \left( {{i = 1},2} \right)}},{\frac{x_{t_{i}}^{''} - x_{t_{i}}^{\prime}}{z^{''} - z^{\prime}} = {\alpha_{t_{i}}^{''}\mspace{31mu} {\left( {{i = 1},2} \right).}}}$

The relationship between α′_(t) _(i) and α″_(t) _(i) is due torefraction and is determined by Snell's law. The first order paraxialapproximation for small incident angles, as commonly used in the opticsliterature. The result is that Snell's refraction law is approximatelyan additive constant independent of the original ray direction.

α″_(t) _(i) ≈α′_(t) _(i) +Δα_(t)(x _(t) _(i) ).

The angle difference Δαt(x_(t) _(i) ) depends only on the normaldirection of fluid object's surface at x′_(t) _(i) and the fluidobject's index of refraction, and is therefore independent of theincoming ray direction α′_(t) _(i) . Because it is assumed that thefluid object does not change shape and index of refraction in the shorttime interval, the difference in angles also remains constant during theshort time interval. Subtracting the first equation in this section fromthe second one results in:

${{\Delta\alpha}_{t} = {{\alpha_{t_{i}}^{''} - \alpha_{t_{i}}^{\prime}} = {\frac{x_{t_{i}}^{''} - x_{t_{i}}^{\prime}}{z^{''} - z^{\prime}} - {\frac{x_{t_{i}}^{\prime} - o}{z^{\prime}}\mspace{31mu} \left( {{i = 1},2} \right)}}}},$

From which for x′_(t) _(i) can be solved for in terms of the othervariables:

${x_{t_{i}}^{\prime} - o} = {{\frac{z^{\prime}}{z^{''}}\left( {x_{t_{i}}^{''} - o} \right)} - {\frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}{{\Delta\alpha}_{t}\left( x_{t_{i}}^{\prime} \right)}\mspace{31mu} {\left( {{i = 1},2} \right).}}}$

Similarly, we can solve for x_(t) _(i) +Δt:

${x_{t_{i}}^{\prime} - o} = {{\frac{z^{\prime}}{z^{''}}\left( {x_{t_{i}}^{''} - o} \right)} - {\frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}{{\Delta\alpha}_{t}\left( x_{t_{i}}^{\prime} \right)}\mspace{31mu} {\left( {{i = 1},2} \right).}}}$

Subtracting the previous two equations from each other yields adifference equation:

${x_{{t_{i} + {\Delta \; t}}|}^{\prime} - x_{t}^{\prime}} = {{- \frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}}\left( {{{\Delta\alpha}_{t_{i} + {\Delta \; t}}\left( x_{t_{i} + {\Delta \; t}}^{\prime} \right)} - {{\Delta\alpha}_{t_{i}}\left( x_{t_{i}}^{\prime} \right)}} \right){\left( {{i = 1},2} \right).}}$

Since x′_(t) ₁ and x′_(t) ₂ are the same point on the fluid object,differences between incidence angles and refraction angles should besame at x′_(t) ₁ and x′_(t) ₂ , that is Δα_(t) _(i) (x′_(t) ₁ )=Δα_(t) ₂(x′_(t) ₂ ). Therefore, subtracting the previous equation with i=2 fromthe previous equation with i=1, yields

${\left( {x_{t_{1} + {\Delta \; t}}^{\prime} - x_{t_{1}}^{\prime}} \right) - \left( {x_{t_{2} + {\Delta \; t}}^{\prime} - x_{t_{2}}^{\prime}} \right)} = {{- \frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}}{\left( {{{\Delta\alpha}_{t_{1} + {\Delta \; t}}\left( x_{t_{1} + {\Delta \; t}}^{\prime} \right)} - {{\Delta\alpha}_{t_{2} + {\Delta \; t}}\left( x_{t_{2} + {\Delta \; t}}^{\prime} \right)}} \right).}}$

The quantity x_(t) ₂ +Δt from the previous equation can then be solvedfor. If x′_(t) ₂ +Δt and x′_(t) ₂ +Δt are also the same point on thefluid object, then the left hand side (LHS) of the previous equation iszero as x′_(t) _(1+Δt) −x′_(t) ₁ =x′_(t) _(2+Δt) −x′_(t) ₂ , and theright hand side (RHS) of the previous equation is also zero as Δα_(t)_(1+Δt) (x′_(t) _(1+Δt) )=Δα_(t) _(2+Δt) (x′_(t) _(2+Δt) ). This showsthat x′_(t) _(2+Δt) =x′_(t) _(1+Δt) −x′_(t) ₁ +x′_(t) ₂ is the solutionto the difference equation above, and therefore x′_(t) ₁ +Δt and x′_(t)₂ +Δt are also the same point on the fluid object.

Finally, it is proven that wiggles at t₁ and t₂ are equal. By thesimilar triangle formula, we have:

$\overset{}{x_{t_{1}}x_{t_{1} + {\Delta \; t}}} = {{\frac{z}{z^{\prime}}\overset{}{x_{t_{1}}^{\prime}x_{t_{1} + {\Delta \; t}}^{\prime}}} = {{\frac{z}{z^{\prime}}\overset{}{x_{t_{2}}^{\prime}x_{t_{2} + {\Delta \; t}}^{\prime}}} = {\overset{}{x_{t_{2}}x_{t_{2} + {\Delta \; t}}}.}}}$

Theorem 2 (Refractive stereo constancy) Suppose there are n≥2 camerasimaging a refractive fluid object, and they are all parallel and closeto each other. Then at any time t, and for any point on the fluidobject, the corresponding wiggle in all the cameras is the same.

FIG. 13 is similar to FIG. 6C but also illustrates quantities relevantto a proof of stereo refractive constancy.

Proof Similar to the previous proof, let x′t be a point on the fluidobject. Tracing rays, at time t, an undistorted ray is emitted from thecenter of the projection o_(j)(j=1, 2) to the point point x′_(t). It isrefracted by the fluid object and intersects the background at x′_(j).At a successive time t+Δt, the fluid object moves, and the light rayfrom the points on the background to the center of the projection nowgoes through points x′_(t+Δt,j) on the fluid object and x_(t) _(i+Δt,j)on the image plane. It will be shown that the wiggle features

$\overset{}{x_{t,1}x_{{t + {\Delta \; t}},1}}\mspace{14mu} {and}\mspace{14mu} \overset{}{x_{t,2}x_{{t + {\Delta \; t}},2}}$

are equal in both views.

As in the previous proof, it is first shown, that x_(t+Δt,1)=x_(t+Δt,2).Following a similar derivation as in the previous proof results in:

${x_{{t + {\Delta \; t}},j}^{\prime} - x_{t}^{\prime}} = {{- \frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}}{\left( {{{\Delta\alpha}_{t + {\Delta \; t}}\left( x_{{t + {\Delta \; t}},j}^{\prime} \right)} - {{\Delta\alpha}_{t}\left( x_{t}^{\prime} \right)}} \right).}}$

Then, subtracting the previous equation with j=2 from the previousequation with j=1, we have:

${x_{{t + {\Delta \; t}},1}^{\prime} - x_{{t + {\Delta \; t}},2}^{\prime}} = {{- \frac{z^{\prime}\left( {z^{''} - z^{\prime}} \right)}{z^{''}}}{\left( {{{\Delta\alpha}_{t + {\Delta \; t}}\left( x_{{t + {\Delta \; t}},1}^{\prime} \right)} - {{\Delta\alpha}_{t + {\Delta \; t}}\left( x_{{t + {\Delta \; t}},2}^{\prime} \right)}} \right).}}$

By the same logic as in the previous proof, whenx′_(t+Δt,1)=x′_(t+Δt,2), both the LHS and RHS of the previous equationare equal to 0. Therefore, x′_(t+Δt,1)=x′_(t+Δt,2) is the solution tothe previous equation.

Thus, x′_(t+Δt,1) and x′_(t+Δt,2) are the same point.

Finally, it can be proven that wiggles from two views are equal:

$\overset{}{x_{{t + {\Delta \; t}},1}x_{t,1}} = {{\frac{z}{z^{\prime}}\overset{}{x_{{t + {\Delta \; t}},1}^{\prime}x_{t}^{\prime}}} = {{\frac{z}{z^{\prime}}\overset{}{x_{{t + {\Delta \; t}},2}^{\prime}x_{t}^{\prime}}} = {\overset{}{x_{{t + {\Delta \; t}},2}x_{{t + {\Delta \; t}},2}}.}}}$

Calculating Fluid Flow Efficiently

It was previously shown that the probabilistic refractive flow methodconsists of two steps. First, the mean {tilde over (v)} and the varianceΣ_(v) of the wiggle features v are solved for from the followingGaussian distribution:

$\overset{}{x_{{t + {\Delta \; t}},1}x_{t,1}} = {{\frac{z}{z^{\prime}}\overset{}{x_{{t + {\Delta \; t}},1}^{\prime}x_{t}^{\prime}}} = {{\frac{z}{z^{\prime}}\overset{}{x_{{t + {\Delta \; t}},2}^{\prime}x_{t}^{\prime}}} = {\overset{}{x_{{t + {\Delta \; t}},2}x_{{t + {\Delta \; t}},2}}.}}}$

To solve for the mean and variance of flow from the previous equation,let the V be the vector formed by concatenating all the optical flowvectors in one frame. That is, V=(. . . , v(x), . . . ). Also, let usrepresent the previous equation in information formP(v|I)=exp(−½V^(T)JV+h^(T)V), where h and J can be calculated from theprevious equation. Then the mean of V is {tilde over (v)}=J⁻¹h andcovariance of V is Σ=J⁻.

In the second step, the fluid flow is calculated by minimizing thefollowing optimization problem based on the mean and variance of thewiggle features computed in the first step.

$\overset{\sim}{u} = {{\arg \; {\min\limits_{u}{\sum\limits_{x}^{\;}{\beta_{1}{{{\frac{\partial\overset{\sim}{v}}{\partial x}u_{x}} + {\frac{\partial\overset{\sim}{v}}{\partial y}u_{y}} + \frac{\partial\overset{\sim}{v}}{\partial t}}}_{\Sigma_{v}}^{2}}}}} + {\beta_{2}\left( {{\frac{\partial u}{\partial x}}^{2} + {\frac{\partial u}{\partial y}}^{2}} \right)} + {\beta_{3}{u}^{2}}}$

Calculating the covariance of each wiggle feature requires inverting theinformation matrix J. This step will be slow if the matrix is large. Toavoid this time consuming inversion, we make a slight change to thefluid flow objective function. Let {tilde over (V)}_(x), {tilde over(V)}_(y), and {tilde over (V)}_(t) be the vectors formed byconcatenating all the partial derivatives of mean wiggle features in aframe, that

${{\overset{\sim}{V}}_{x} = \left( \mspace{14mu} {\ldots \mspace{14mu},{\frac{\partial v}{\partial x}(x)},\ldots}\mspace{14mu} \right)},{{\overset{\sim}{V}}_{y} = \left( \mspace{14mu} {\ldots \mspace{14mu},{\frac{\partial v}{\partial y}(x)},\ldots}\mspace{14mu} \right)},{and}$${\overset{\sim}{V}}_{t} = {\left( \mspace{14mu} {\ldots \mspace{14mu},{\frac{\partial v}{\partial t}(x)},\ldots}\mspace{14mu} \right).}$

Similarly, let U_(x), U_(y) be the vectors formed by concatenating allthe x-components and y-components of u in a frame respectively. Then therefractive flow can be calculated as follows:

${\min\limits_{U}{{\beta_{1}\left( {{{\overset{\sim}{V}}_{x} \cdot U_{x}} + {{\overset{\sim}{V}}_{y} \cdot U_{y}} + {\overset{\sim}{V}}_{t}} \right)}^{T}{J\left( {{{\overset{\sim}{V}}_{x} \cdot U_{x}} + {{\overset{\sim}{V}}_{y} \cdot U_{y}} + {\overset{\sim}{V}}_{t}} \right)}}} + {\beta_{2}\left( {{{D_{x}U_{x}}}^{2} + {{D_{y}U_{x}}}^{2} + {{D_{x}U_{y}}}^{2} + {{D_{y}U_{y}}}^{2}} \right)} + {\beta_{3}{U}^{2}}$

where D_(x) and D_(y) are the partial derivative matrices to x and yrespectively. The smoothness term of the previous equation is exactlythe same as that in the equation for ũ above, and the data term of theequation for ũ above is

(V _(x) ·U _(x) +V _(y) ·U _(y) +V _(t))^(T) J(V _(x) ·U _(x) +V _(y) ·U_(y) +V _(t))=∥V _(x) ·U _(x) +V _(y) ·U _(y) +V _(t)∥_(J-1) =∥V _(x) ·U_(x) +V _(y) ·U _(y) +V _(t)∥_(Σ),

which is also similar to the data term in the equation for ũ aboveexcept that it jointly considers all the wiggle vectors in a frame.Therefore, this change will not affect the result too much, but themethod is more computationally efficient, as we never need to computeJ⁻¹. The term never appears in the refractive flow expression above.

Probabilistic Interpretation in the Refractive Stereo

In this section, it is shown that the data term defined in Section 5 forrefractive stereo is equal to the negative log of the conditionalmarginal distribution. Let v_(R)(x)˜N(v _(R)(x),Σ_(R)(x)) andv_(L)(x+d(x))˜N(v _(L)(x+d(x)), Σ_(L)(x+d(x))) be the optical flow fromthe left and right views, where v _(L) and v _(R) are the means of theoptical flow and and Σ_(L) and Σ_(R) are the variances of optical flow.Then the data term of the refractive stereo is

${f\left( {v_{R},v_{L}} \right)} = {{{- \log}\; {{cov}\left( {v_{L},v_{R}} \right)}} = {{{- \log}{\int_{v}{{P_{L}(v)}{P_{R}(v)}{dv}}}} = {{{- \log}{\int_{v}{{N\left( {{v;{\overset{\_}{v}}_{L}},\Sigma_{L}} \right)}{N\left( {{v;{\overset{\_}{v}}_{R}},\Sigma_{R}} \right)}{dv}}}} = {{\frac{1}{2}\log {{\Sigma_{L} + \Sigma_{R}}}} + {\frac{1}{2}{{{\overset{\_}{v}}_{R} - {\overset{\_}{v}}_{L}}}_{\Sigma_{L} + \Sigma_{R}}^{2}} + {const}}}}}$

Where N(v; v _(L), Σ_(L)) is shorthand notation for the Gaussiandistribution probability density function

${N\left( {{v;\overset{\_}{v}},\Sigma} \right)} = {\frac{1}{2\pi \sqrt{\Sigma }}{\exp \left( {{- \frac{1}{2}}{\overset{\_}{v}}^{T}\Sigma^{- 1}\overset{\_}{v}} \right)}}$

Recall that the optical flow calculated by the method is degraded bynoise. Specifically, let v(x) be the ground truth optical flow from theright view at x. The mean optical flow from the right view (or leftview) calculated by the method equals the ground truth optical flow plusGaussian noise with zero-mean and variance equal to Σ_(R)(or Σ_(L)),that is:

P( v _(R)(x)|v(x))=N( v _(R)(x); v(x), Σ_(R)(x)), P( v_(L)(x+d(x))|v(x), d(x))=N( v _(L)(x+d(x)); v(x), Σ_(L)(x+d(x)))

To evaluate the probability of d, let us consider the marginaldistribution

${P\left( {{{\overset{\_}{v}}_{R}(x)},\left. {{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)} \middle| {d(x)} \right.} \right)} = {{\int_{v}{{P\left( {{{\overset{\_}{v}}_{R}(x)},{{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)},\left. {v(x)} \middle| {d(x)} \right.} \right)}{{dv}(x)}}} = {{\int_{v}{{P\left( {{\overset{\_}{v}}_{R}(x)} \middle| {v(x)} \right)}{P\left( {\left. {{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)} \middle| {v(x)} \right.,{d(x)}} \right)}{P(v)}{{dv}(x)}}} = {\int_{v}{{N\left( {{{{\overset{\_}{v}}_{R}(x)};{v(x)}},{\Sigma_{R}(x)}} \right)}{N\left( {{{{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)};{v(x)}},{\Sigma_{L}\left( {x + {d(x)}} \right)}} \right)}{P(v)}{{dv}(x)}}}}}$

Assuming that P(v) has an uniform prior, we have:

${\log \; {P\left( {{{\overset{\_}{v}}_{R}(x)},\left. {{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)} \middle| {d(x)} \right.} \right)}} = {{\log {\int_{v}{{N\left( {{{{\overset{\_}{v}}_{R}(x)};v},{\Sigma_{R}(x)}} \right)}{N\left( {{{{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)};v},{\Sigma_{L}\left( {x + {d(x)}} \right)}} \right)}{P(v)}{dv}}}} = {{{\log {\int_{v}{{N\left( {{{{\overset{\_}{v}}_{R}(x)};v},{\Sigma_{R}(x)}} \right)}{N\left( {{{{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)};v},{\Sigma_{L}\left( {x + {d(x)}} \right)}} \right)}{dv}}}} + {const}} = {{{\log {\int_{v}{{N\left( {{v;{{\overset{\_}{v}}_{R}(x)}},{\Sigma_{R}(x)}} \right)}{N\left( {{v;{{\overset{\_}{v}}_{L}\left( {x + {d(x)}} \right)}},{\Sigma_{L}\left( {x + {d(x)}} \right)}} \right)}{dv}}}} + {const}} = {{- {f\left( {v_{R},v_{L}} \right)}} + {const}}}}}$

Therefore, the data term is equal to the negative log of conditionalmarginal distribution (plus a constant).

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. An augmented reality apparatus comprising: avideo camera with an imaging plane, the video camera configured tocapture video of a scene including a textured background, the videoincluding captured light that passes from the textured background to theimaging plane through a refractive field; and an augmented realitydisplay configured to render the video of the scene with a visualizationof actual motions of the refractive field, the actual motions determinedby correlating, over time, representations of apparent motions of thetextured background observed at the imaging plane in the video fromframe to frame by modelling translations of the refractive field ascausing the apparent motions.
 2. The apparatus of claim 1, wherein thevisualization includes representations of velocities of the refractivefield based upon the correlated representations of the apparent motions.3. The apparatus of claim 1, wherein the refractive field is a fluid. 4.The apparatus of claim 1, further including a processor configured todetermine the actual motions by correlating the representations of theapparent motions through an assumption of frame-to-frame changes in atleast one of intensity and phase resulting from the motions alone. 5.The apparatus of claim 1, further including a processor configured todetermine the actual motions by correlating the representations of theapparent motions using an approximation that the apparent motions areconstant from one video frame to another video frame.
 6. The apparatusof claim 1, wherein the visualization of actual motions of therefractive field is a visualization of a fluid turbulence, updraft,downdraft, or vortex.
 7. The apparatus of claim 1, wherein the actualmotions of the refractive field arise from motion of one or morerefractive objects in a fluid, and wherein the representations of theapparent motions are representations of motions of the one or morerefractive objects.
 8. The apparatus of claim 7, wherein the one or morerefractive objects include at least one of a smoke particle, molecule,region of the fluid differing in temperature from a surrounding region,region of fluid differing in pressure from a surrounding region,droplet, bubble, exhaust, hydrocarbon, or volatile substance.
 9. Theapparatus of claim 1, wherein the textured background is a naturalbackground.
 10. The apparatus of claim 1, wherein the display iswearable and further configured to render the video of the scene with avisualization of actual motions of the refractive field for viewing by awearer's eye.
 11. The apparatus of claim 10, further comprising aprocessor configured to determine the actual motions.
 12. The apparatusof claim 10, wherein the video camera and the augmented reality displayform part of an air traffic control system, an onboard airplane system,a weather monitoring system, a leak detection system, a civilengineering system, an aeronautical engineering system, or a ballisticsor combustion monitoring system.
 13. A wearable device comprising: avideo camera with an imaging plane, the video camera configured tocapture video of a scene including a textured background, the videoincluding captured light that passes from the textured background to theimaging plane through a refractive field; and an augmented realitydisplay configured to render a visualization of actual motions of therefractive field in a wearer's eye, the actual motions determined bycorrelating, over time, representations of apparent motions of thetextured background observed at the imaging plane in the video fromframe to frame by modelling translations of the refractive field ascausing the apparent motions.
 14. A video microscope comprising: a videocamera with a microscopic field of view and an imaging plane, the videocamera configured to output video by capturing light that passes from atextured background to the imaging plane through a refractive field inthe microscopic field of view; and a display configured to show avisualization of actual motions of the refractive field in themicroscopic field of view, the actual motions determined by correlating,over time, representations of apparent motions observed at the imagingplane in the video from frame to frame by modelling translations of therefractive field as causing the apparent motions.
 15. The videomicroscope of claim 14, wherein the visualization includesrepresentations of velocities of the refractive field based upon thecorrelated representations of the apparent motions.
 16. The videomicroscope of claim 14, wherein the refractive field is a fluid.
 17. Thevideo microscope of claim 14, further including a processor configuredto determine the actual motions by correlating the representations ofthe apparent motions using an assumption of frame-to-frame changes in atleast one of intensity and phase resulting from the motions alone. 18.The video microscope of claim 14, further including a processorconfigured to determine the actual motions by correlating therepresentations of the apparent motions using an approximation that theapparent motions are constant from one video frame to another videoframe.
 19. The video microscope of claim 14, wherein the motions of therefractive field arise from motion of one or more refractive objects ina fluid, and wherein the representations of the apparent motions arerepresentations of motions of the one or more refractive objects. 20.The apparatus of claim 19, wherein the one or more refractive objectsinclude at least one of a molecule, region of the fluid differing intemperature from a surrounding region, region of fluid differing inpressure from a surrounding region, droplet, bubble, hydrocarbon, orvolatile substance.
 21. The video microscope of claim 14, wherein thetextured background is a natural background.
 22. The video microscope ofclaim 14, wherein the textured background is an artificial texturedbackground having microscopic features.
 23. The video microscope ofclaim 14, wherein the microscopic field of view includes at least apartial view of a chemical sample or a cell culture or other biologicalsample.